Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: HDF5 for .NET Standard #57

Open
michaelosthege opened this issue Jun 11, 2016 · 24 comments
Open

Feature request: HDF5 for .NET Standard #57

michaelosthege opened this issue Jun 11, 2016 · 24 comments

Comments

@michaelosthege
Copy link

For broad compatibility and less fragmentation it would be really great to have a .NET Standard implementation of HDF5. This table gives a good overview which versions of Mono/UWP/.NET Framework work with .NET Standard: https://github.com/dotnet/corefx/blob/master/Documentation/architecture/net-platform-standard.md#mapping-the-net-platform-standard-to-platforms

As I understand it, it would require to build the HDF5 source with VS into a PCL.

@dynamatt
Copy link

I've just had a play around with this. One of the main differences with the .NET Standard is how DLL's are loaded. I think the biggest challenge here will be resolving the DLL's.

I managed to convert the project and get the unit tests to pass but that was only after manually copying the HDF5 DLL's into the bin folder.

I will continue to look into this and try and learn how DLL loading is done by the .NET Standard.

@michaelosthege
Copy link
Author

For reference, a FAQ on .NET Standard: https://github.com/dotnet/standard/blob/master/docs/faq.md

Unfortunately my knowledge of HDF5 is restricted to very high-level interfacing using h5py in Python.. But I do believe that a .NET Standard HDF5 API would be a good long-term investment.

@aloneguid
Copy link

I think it might be a completely new project. This one is P/Invoking to a windows dll which is extremely platform dependent. In order to properly migrate to .NET Standard (so that works on phones, windows, Linux, mac etc.) you'll have to rewrite the library to natively use HDF5 format.

@michaelosthege
Copy link
Author

If I#m not mistaken, P/Invoke can also be done on UWP.

Would it be possible to build targeting UWP and add this to the NuGet package?

@JanWosnitza
Copy link
Collaborator

This one is P/Invoking to a windows dll which is extremely platform dependent.

At the moment it is, true. But as you can see in HDF5/H5DLLImporter.cs there was at least an attempt to support Linux and OSX too.
Would be awesome if the library was platform independent since HDF5 is already.

@michaelosthege
Copy link
Author

I just threw that code into a fresh UWP project. Here's the thing: the System.Environment.OSVersion API is not supported on UWP. (Also see this thread.)
But DllImport does work, so I assume it should work. But I don't understand enough of how the HDF5 sources are built in order to build them for UWP.

@JanWosnitza
Copy link
Collaborator

JanWosnitza commented Dec 22, 2016

HDF5 is build with CMake. CMake is able to generate a Visual Studio solution-file. When you open the solution in VS you can build the project "libraries/hdf5-shared" which will generate the desired .dlls. Should work out of the box. If you need help with CMake tell me ;)

image

@michaelosthege
Copy link
Author

@JanWosnitza oh, I didn't notice that there was a reply!
Thanks - I will definitely try to do it!
(some time in a few weeks..)

@michaelosthege
Copy link
Author

So I'm trying to build it right now. My progress so far:

  1. downloaded CMake-hdf5-1.8.18.zip from here
    2. ran build-VS2015-64.bat that generated a build directory containing a VS solution
  2. opened the solution in VS 2015 - building for x86 worked
  3. created an ARM solution platform with the Configuration Manager
  4. in the project properties of hdf5-shared, H5detect and h5make_libsettings unter Linker/All Options/Additional Options the %(AdditionalOptions) /machine:X86 had to be changed to %(AdditionalOptions) /machine:ARM
  5. with that I got a successful build in Release/ARM mode (hdf5.dll)
  6. now the same with libraries/hl/hdf5_hl-shared and now I also have a hdf5_hl.dll

So far so good. I'm actually surprised I made it that far. But what's next? I suppose these should end up in HDF.PInvoke\native\HDF5 1.8\ARM? But what about the PDBs and szip/zlib/msvcr120/msvcp120 ?

Release ARM.zip

What would it take to get them distributed with the HDF.PInvoke package?

thanks!

@gheber
Copy link
Member

gheber commented Feb 12, 2017

Do the tests pass? I'd be happy to include them, but I don't have an ARM system for testing. Links to the source code for SZIP and ZILB can be found near the bottom of this page. I think we decided against distributing msvcr120/msvcp120 & Co.

@michaelosthege
Copy link
Author

I was able to build SZIP and ZLIB with Release/ARM, but no .dll output was generated.

The PInvoke code is not entirely compatible with UWP since it does use some APIs that were depreceated. I created a new project for both PInvoke and the UnitTests and right now I'm trying to make it work for 86x UWP.
After that I'll try to PInvoke the ARM dlls that came out of my previous comment. But I do not understand if missing szip.dll and zlib.dll is a problem.

@gheber
Copy link
Member

gheber commented Feb 13, 2017

Which APIs were deprecated? (Just curious.)

SZIP and ZLIB aren't an issue as long as you don't try to read or write compressed (w/ deflate or szip) datasets.

@michaelosthege
Copy link
Author

No success yet, but some insights:
UWP does not support changing the Environment variables, therefore the DllImport has to work in a different way. I tried this:

#if X86
        const string Platform = "x86";
#elif X64
        const string Platform = "x64";
#elif ARM
        const string Platform = "ARM";
#endif

        const string DLLFilePath = "\\HDF.PInvoke.UWP\\native_1.8\\" + Platform + "\\hdf5.dll";
        const string HLDLLFilePath = "\\HDF.PInvoke.UWP\\native_1.8\\" + Platform + "\\hdf5_hl.dll";

I think the conditional compilation symbols are easier to maintain than the current implementation based on the PATH variable.

Or is there another reason why the PATH variable has to be set?

@michaelosthege
Copy link
Author

@gheber

  • Reflection/Assembly > can't get the path to the dll
  • no access to Environment variables
  • Thread is now substituted with System.Threading.Tasks.Task (had to modify some UnitTests)

I think overall these are no big deal as long as one can get the DllImport to work with const relative paths.

@michaelosthege
Copy link
Author

michaelosthege commented Feb 13, 2017

Another update:
I was able to set up a .NET Standard 1.1 library with a modified Constants.cs:

#if X86
        const string DLLFolder = "bin32";
#elif X64
        const string DLLFolder = "bin64";

#elif ARM
        const string DLLFolder = "ARM";
#endif
        public const string DLLFileName = DLLFolder + "\\" + "hdf5.dll";

        public const string HLDLLFileName = DLLFolder + "\\" + "hdf5_hl.dll";

Several Attributes had to be slightly modified too.

Within the solution, I can reference it and all tests pass! (tested x86/x64)

A NuGet package with embedded PowerShell script to copy the right dlls to the output directory seems to be the best solution.
(The DLLs can also be EmbeddedResources, but in .NET Standard 1.1 there's no FileIO to save them..)

In short: It was relatively easy to make it into a .NET Standard 1.1 library, but creating the right NuGet package is tricky.

Here's what I have: https://1drv.ms/u/s!Ao_tF5GvDxQdv45-snwBuoh5qQhnMg

@Blealtan
Copy link

Blealtan commented Jul 1, 2017

Actually DllImport SHOULD work fine in any implementation that supports .Net Standard, always. It's a required interface, and should work with anything like a shared object or a DLL; in fact, .Net Core handles this pretty well. Just remove the ".dll" in your constant, tune directory prefixes as @michaelosthege did, and it'll work out of the box.

There's a good example dealing with cross-platform P/Invoke: ManagedBass. It targets at AnyCPU and simply refers to bass, works well except that its user has to get bass.dll by themselves. Assuming you'd prefer ship h5 DLLs via NuGet, maybe the best choice for you is to publish a series of packages, each containing h5 DLLs for one single platform, like windows, uwp, Linux, and so on. But this is to be a hard work; I'm not so familiar with this, but if you want, I think I can help.

As a .Net Core heavy user, I'm eager to see more and more projects (and also NuGet packages) become platform independent.

@Blealtan
Copy link

Blealtan commented Jul 1, 2017

Sorry but I've forget the global variables.. It seems that being totally platform independent is impossible due to this design of original HDF5 library. However, if possible, distribute runtime packages separately is still a good practice, just like many Microsoft's own packages (I mean those packages' names start with runtime.) does.

@hpbieker
Copy link

Any progress here?

@surban
Copy link

surban commented Nov 11, 2017

See my work at #127. I have NuGet packages at https://www.myget.org/feed/Packages/coreports .

@hpbieker
Copy link

Thank you, any chance to get it published on the official nuget repository?

@hokb
Copy link
Collaborator

hokb commented Jan 10, 2018

It sounds compelling to target .NET Standard 1.?, indeed. However, the statement "this library targets .NET Standard" implies that any user of the library can reference the library and be sure that is works on any runtime / platform implementing the same version of .NET Standard - even on any platform not known so far!

Now, since HDF/PInvoke is basically a thin wrapper around the HDF5 binaries this statement will not hold. HDF/PInvoke derives the platform dependence from the unmanaged binaries it wraps. According to the approach followed above in this thread, a user - while being able to reference the library as .NET Standard lib from any project - will still have to deal with the unmanaged dependencies for the specific target platform.

So there will be no full platform compatibility anyways. There is no (feasible) way to create an assembly for Windows, take the .exe and run it on iOS and expect it to work when unmanaged binaries are involved (and leaving too exotic solutions aside). At development time a conscious decision about the target platform is required.

A feasible solution for now might be to
• Provide & maintain project files for the most popular runtimes (.NET Framework, .NET Core, Xamarin, mono, … ?). To target specific runtimes instead of .NET Standard is IMO more applicable here.
• Leave it to the developer (user of HDF/PInvoke) to deal with unmanaged binaries, except for the most popular platforms which might be provided in a more 'official' manner (?), and
• Collect recommendations and experiences for HDF/PInvoke on individual platforms in a wiki. Maybe there will be a more unified solution in the future.

@Apollo3zehn
Copy link

Apollo3zehn commented Jan 17, 2018

As already stated in the HDF mailing list I would also love to see a solution to be able to use a (in my case) .NET Core project which references the HDF.PInvoke library without giving warnings about possible incompatibility. If I am not wrong, the upcoming .NET Core 2.1 will be the last version that allows loading NET Framework assemblies.

I am not sure if targetting specific runtimes would solve the problem. In my opinion this leads to more overhead than required because multiple projects with marginal differences must be maintained then. It is not clear to me how one would ensure that the right native library is loaded (and distributed). Manually copying the correct library would at least for me no realistic option. I am planning to use the NuGet.PackageManagement package to let the end user install and uninstall plugins (where one of the plugins depends on HDF). This requires that all packages (or plugins) and their dependencies are fully functional when they are downloaded. This process should not require the end user to manually download appropriate native libaries from a web page.

If instead say for example the following .NET Standard packages would exist (like @Blealtan suggested):

  • HDF.PInvoke.Windows
  • HDF.PInvoke.Linux
  • ...

Then they could deliver the right native dependencies and it is clear to the user that the Windows package does not work on Linux. I am sure that using msbuild and a modified .targets file the whole build and publish process could be automated.

Microsoft itself also publishes platform specific packages that target (among others) .NET Standard. See for example System.Diagnostics.EventLog. Unfortunately this package does not make clear it is only working on Windows. But with proper naming as suggested above, it should be unambiguous.

@Apollo3zehn
Copy link

Please see my comment for new developments regarding HDF.PInvoke for .NET Standard.

@PhilPJL
Copy link

PhilPJL commented Apr 13, 2022

Why is this issue open when I'm using this
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests