Hello Jon,
You are correct. I was able to pass the pFilterTableNameIn as string by setting the CharSet=CharSet.Unicode in the function declaration. Thank you for pointing this out. My p/invoke(s) now have the following form,
[DllImport("stdmdlbltin.dll", CharSet = CharSet.Unicode)] public static extern IntPtr mdlFilterIterator_create(int pModelRefIn, string pFilterTableNameIn);
With mdlFilter_getName however, I was not able to pass the pFilterNameOut parameter as ref (nor as out). The program hangs and eventually forces me to abort MicroStation. When I realized that pFilterNameOut was ByVal in the VBA wrapper I removed "ref" and the program ran without a crash. Of course, it does not change the value of pFilterNameOut because the parameter is not passed as a reference.
There are a few other functions that have the same issue - mdlFilter_getDescription, mdlFilterMember_getName, and mdlFilterMember_getDescription. Compare this with other functions like mdlProjection_perpendicular() where the first three parameters have a MODE=OUT and the VBA wrapper passes these ByRef.
As a work around I was able to use mdlFilter_getPathName. Turns out this is a better solution for my case.