dotnet ef migrations fail when using custom output paths and separate project/startup-projects

[chris-garrett]

dotnet ef migrations fail when using custom output paths and separate project/startup-projects

Hi folks,

I'm trying to get my projects setup in docker and I'm running into this error:

error MSB4057: The target "GetEFProjectMetadata" does not exist in the project.

I used Brice's suggestion here #23691 (comment) and it works but only when everything is in a single project.

Other issues that are similar but I'm either missing a step or this is broken.
#23691
#12220

Example repo

I have a repo with 4 scenarios here: https://github.com/chris-garrett/EfTools. I removed docker for the time being since this is reproducible without it.

1. d180df5 Everything in one project with no modifications to output paths: Works
2. 7f0c9b3 Everything in one project with modifications to output paths ([obj|bin]/containers): Works
3. 1fb088e Separate projects for contexts / migration tool with no modifications to output paths: Works
4. 5c264bc Separate projects for contexts / migration tool with modifications to output paths ([obj|bin]/containers): Does not work

Steps

# nuke any bin/obj folders
$ ./tools-nuke.sh

# create the migration
$ ./tool-create-migration.sh InitialMigration

# run the migration and verify its working
$ ./tool-migrate.sh

Verbose output

$ ./tool-create-migration.sh InitialMigration
Using project './EfTools.Data.csproj'.
Using startup project '../EfTools.Migrations/EfTools.Migrations.csproj'.
Writing 'obj/container/EfTools.Data.csproj.EntityFrameworkCore.targets'...
dotnet msbuild /target:GetEFProjectMetadata /property:EFProjectMetadataFile=/tmp/tmpm1K4cl.tmp /verbosity:quiet /nologo ./EfTools.Data.csproj
Writing 'obj/container/EfTools.Migrations.csproj.EntityFrameworkCore.targets'...
dotnet msbuild /target:GetEFProjectMetadata /property:EFProjectMetadataFile=/tmp/tmpKnaJFl.tmp /verbosity:quiet /nologo ../EfTools.Migrations/EfTools.Migrations.csproj                                                                                                                     
/mnt/nvme0/projects/dotnet/eftools/EfTools.Migrations/EfTools.Migrations.csproj : error MSB4057: The target "GetEFProjectMetadata" does not exist in the project.                                                                                                                           
Microsoft.EntityFrameworkCore.Tools.CommandException: Unable to retrieve project metadata. Ensure it's an SDK-style project. If you're using a custom BaseIntermediateOutputPath or MSBuildProjectExtensionsPath values, Use the --msbuildprojectextensionspath option.                     
   at Microsoft.EntityFrameworkCore.Tools.Project.FromFile(String file, String buildExtensionsDir, String framework, String configuration, String runtime)                                                                                                                                  
   at Microsoft.EntityFrameworkCore.Tools.RootCommand.Execute(String[] _)                                                                     
   at Microsoft.EntityFrameworkCore.Tools.Commands.CommandBase.<>c__DisplayClass0_0.<Configure>b__0(String[] args)                            
   at Microsoft.DotNet.Cli.CommandLine.CommandLineApplication.Execute(String[] args)                                                          
   at Microsoft.EntityFrameworkCore.Tools.Program.Main(String[] args)                                                                         
Unable to retrieve project metadata. Ensure it's an SDK-style project. If you're using a custom BaseIntermediateOutputPath or MSBuildProjectExtensionsPath values, Use the --msbuildprojectextensionspath option. 

Provider and version information

EF Core version: 5.0.1
Database provider: Microsoft.EntityFrameworkCore.Sqlite
Target framework: .NET 5.0.101
Operating system: Ubuntu 20.04.1 LTS
IDE: command line

[ajcvickers]

/cc @bricelam

[bricelam]

Looks like you have that pesky sample code I tried to get rid of in dotnet/dotnet-docker#1033...

[bricelam]

it works but only when everything is in a single project

Hmm, yep. The tools currently assume both projects have the same MSBuildProjectExtensionsPath...

[bricelam]

As part of #18840, we should explore ways we can improve this experience. Probably worth adding some docker-specific guidance in the docs too.

[Nefarion]

As a quick fix, it would be good to pass runtime, framework, and configuration to

efcore/src/dotnet-ef/RootCommand.cs

Line 75 in 901bfb4

var project = Project.FromFile(projectFile, _msbuildprojectextensionspath!.Value());

as done directly below

efcore/src/dotnet-ef/RootCommand.cs

Lines 76 to 81 in 901bfb4

	var startupProject = Project.FromFile(
	startupProjectFile,
	_msbuildprojectextensionspath.Value(),
	_framework!.Value(),
	_configuration!.Value(),
	_runtime!.Value());

This would pick up on "dynamic" TargetPaths already, and not fail if Projects produce e.g. <ProjectName>.Debug.dll or <ProjectName>.Release.dll depending on Configuration

[billybraga]

We have this problem in our project. We have a separate project for our EF and API. We want to use a different build folder for linux and windows to allow us to build using wsl on the same folder as windows. We added this to our Directory.Build.Props:

        <OsKey Condition="$([MSBuild]::IsOSPlatform('Linux'))">linux</OsKey>
        <OsKey Condition="$([MSBuild]::IsOSPlatform('OSX'))">osx</OsKey>
        <OsKey Condition="$([MSBuild]::IsOSPlatform('Windows'))">win</OsKey>
        <OutputPath>bin/$(OsKey)/$(Configuration)</OutputPath>
        <BaseIntermediateOutputPath>obj/$(OsKey)</BaseIntermediateOutputPath>

But since --msbuildprojectextensionspath is applied to both project and startupProject in RootCommand.cs (and without env var replacement or possibility of resolving it relative to the project), we can't use our custom build folders.

I think we should be allowed to use relative paths for --msbuildprojectextensionspath so we could use --msbuildprojectextensionspath obj/win to have EF resolve it per project.

[georg-jung]

Was this postponed because the effort required did not justify the implementation, or the implementation was too complicated, or because the priority was too low?

I ask because this affects almost all projects I do with EF Core. My typical use case is:

I split my code into several csprojs:

• one for the mvc/blazor/desktop/.... UI app, "X.Web"
• one for the context and migrations, "X.Data"
• one for the models of the context, "X.Model".

I regularly choose this approach because I think it has some advantages (if you think this choice is sub-par, let me know):

• If I have a separate project for tooling/a nightly cronjob/maintenance/.... I can just depend on the same data access code
• each project can depend only on the packages it is supposed to depend on, ensuring clean separation at compile time (no one can accidentally use view-specific types in models, etc.; the X.Model could even be netstandard with Nullable if needed)

I also redirect my bin/ and obj/ folders to be at the root of the repo, similar to what @AArnott does in Library.Template. I think this has these advantages:

• There is a constant path that contains the build artifacts. This makes it easy to collect them in DevOps pipelines and simplifies pipeline configuration.
• Sometimes I want to manually delete the obj/ content if something goes wrong. This is really easy if obj/ is in the root of the repo.
• I find it clean not to clutter the repo directory with build artifacts, but to have them in an easy to find place.

Because of this issue, the dotnet ef tooling breaks. I considered several options:

1. Just use the Visual Studio's ef tooling. This works, but doesn't really feel right. It makes the ef commands unavailable in DevOps. So I could not create a migrationbundle as part of my CI/CD. It also excludes all non-VS/non-Windows contributors, leaving them with hard to understand error messages.
2. Don't redirect bin/ and obj/. Lose the benefits mentioned above.
3. Only redirect bin/, not obj/. Works, but feels wrong. Hard to understand later or for new contributors. Clutters the repo directory with build artifacts in a rather strange way.
4. Wait until this issue is fixed.

Obviously I was hoping for option 4 ;-).

If you think what I'm doing is somehow weird or wrong or could be done better, please let me know. I would have thought this was a typical scenario for projects beyond very simple ones, but maybe I'm wrong when I look at the votes for this issue.

If you think that would be an option, I could try to create a PR for this, but you can probably better tell whether that makes sense to try.

[AgentEnder]

Hey @bricelam, I hate to just come in with a "bump", as a maintainer on a large OSS project myself I get that is not ideal...

That said, is there a status update of some kind on this issue? Aside from my work on Nx itself, I am the lead maintainer for the .NET plugin for it, and we redirect outputs into a folder that isn't within the project root as a standard. We do so to better fit Nx's conventions, and while its optional I'd like to continue to do that...

Nx has remote caching, and distributed task execution. These features combined lets you build a parent project (say your webapi), and kick off child projects building (class libraries, etc) on any number of machines. As dependant builds finish, and parent builds start, the build intermediates and results are downloaded onto the machine that will be running the parent build.

For large projects this is much, much faster.

If the intermediates folder is within the project root, specifying the cache locations and task inputs for build targets is a bit more complex than I'd like.

Is there anything I may be able to contribute which could further push this? My main reason for coming in with the bump is that we have the "punted-for7.0" tag, and 7.0 has released.

---

For reference, this is what our users end up running into: nx-dotnet/nx-dotnet#673

[TheDevelolper]

I wrote an article with a workaround for Nx.

https://dev.to/simplifycomplexity/using-entityframework-in-a-nxdotnet-environment-cfb

Though the problem lies with EF core not respecting the build config xml file.

I'll update the article to reference this issue.

[genadytr]

@bricelam Any chance this issue will be addressed soon?

[bricelam]

I hope to look at it in the 8.0 timeframe.

[benpsnyder]

I wrote an article with a workaround for Nx.

https://dev.to/simplifycomplexity/using-entityframework-in-a-nxdotnet-environment-cfb

Though the problem lies with EF core not respecting the build config xml file.

I'll update the article to reference this issue.

I think I have a better temporary workaround for an Nx workspace. I have not tested yet for side-effects, but here is my solution. @AgentEnder you may be interested.

I simply renamed me Directory.Build.props to Directory.Build.Config.props

Then I conditionally excluded my entity-framework-core library in a fresh Directory.Build.props

And I used a custom command called ef-migrate on my library's project.json

[sharky98]

@bricelam 8.0 being released now, any updates?

[ajcvickers]

@sharky98 This issue is in the Backlog milestone. This means that it is not planned for the next release. We will re-assess the backlog following the this release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources. Make sure to vote (👍) for this issue if it is important to you.

[Inurias]

I run into this issue regularly. I also redirect my /bin and /obj via

<BaseOutputPath>$(MSBuildThisFileDirectory)artifacts\bin\$(MSBuildProjectName)\</BaseOutputPath>
<BaseIntermediateOutputPath>$(MSBuildThisFileDirectory)artifacts\obj\$(MSBuildProjectName)\</BaseIntermediateOutputPath>

Until this is fixed, is there any known way/workaround to use dotnet ef migrations with redirected /bin and /obj paths?

[aradalvand]

Please consider adding this to the v9.0 milestone; this has been a problem for years.
This basically makes EF Core incompatible with the new artifacts output layout.

Any workarounds in the meantime?

[vchirikov]

read the issue, workaround is to use the --msbuildprojectextensionspath artifacts/obj/<ProjectName> flag of dotnet ef

[aradalvand]

@vchirikov It's incredibly ironic to tell other people to "read the issue" and simultaneously demonstrate not having done so yourself — that workaround does not work if you have separate startup and migration projects: see literally the original post here and also here, as well as this other comment.

[vchirikov]

.gitignore:

obj/

Directory.Build.targets:

<Project>
  <!-- To use dotnet ef without 'msbuildprojectextensionspath' -->
  <Import Condition="Exists('$(MSBuildProjectDirectory)\obj\$(MSBuildProjectFile).EntityFrameworkCore.targets')" Project="$(MSBuildProjectDirectory)\obj\$(MSBuildProjectFile).EntityFrameworkCore.targets" />
</Project>

and use dotnet ef as usual

[paulomorgado]

I have a DbContext.csproj that defines my DbContext and a Startup.csproj that has the host, using UseArtifactsOutput=true.

I've set environment variables MSBuildDebugEngine=1 and MSBUILDDEBUGPATH=<DIRECTORY>, a per https://github.com/dotnet/msbuild/blob/main/documentation/wiki/MSBuild-Environment-Variables.md and was able to collect binlogs for the builds dotnet ef invoke.

When I invoke dotnet ef dbcontext --project DbContext\DbContext.csproj --startup-project Startup\Startup.csproj --msbuildprojectextensionspath artifacts\obj\DbContext I get the following error:

...\Startup\Startup.csproj : error MSB4057: The target "GetEFProjectMetadata" does not exist in the project. Unable to retrieve project metadata. Ensure it's an SDK-style project. If you're using a custom BaseIntermediateOutputPath or MSBuildProjectExtensionsPath values, Use the --msbuildprojectextensionspath option.

Checking the binlog for the build for DbContext.csproj, I can see that it succeeded and artifacts\obj\DbContext\DbContext.csproj.EntityFrameworkCore.targets file is there with the GetEFProjectMetadata target.

However, for Startup.csproj, there is no artifacts\obj\Startup\Startup.csproj.EntityFrameworkCore.targets file and, thus, no GetEFProjectMetadata target, which causes the build to fail.

Because the file is not there, there is no file to import.

I've created a local EntityFrameworkCore.targets with this:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="GetEFProjectMetadata">
    <MSBuild Condition=" '$(TargetFramework)' == '' "
             Projects="$(MSBuildProjectFile)"
             Targets="GetEFProjectMetadata"
             Properties="TargetFramework=$(TargetFrameworks.Split(';')[0]);EFProjectMetadataFile=$(EFProjectMetadataFile)" />
    <ItemGroup Condition=" '$(TargetFramework)' != '' ">
      <EFProjectMetadata Include="AssemblyName: $(AssemblyName)" />
      <EFProjectMetadata Include="Language: $(Language)" />
      <EFProjectMetadata Include="OutputPath: $(OutputPath)" />
      <EFProjectMetadata Include="Platform: $(Platform)" />
      <EFProjectMetadata Include="PlatformTarget: $(PlatformTarget)" />
      <EFProjectMetadata Include="ProjectAssetsFile: $(ProjectAssetsFile)" />
      <EFProjectMetadata Include="ProjectDir: $(ProjectDir)" />
      <EFProjectMetadata Include="RootNamespace: $(RootNamespace)" />
      <EFProjectMetadata Include="RuntimeFrameworkVersion: $(RuntimeFrameworkVersion)" />
      <EFProjectMetadata Include="TargetFileName: $(TargetFileName)" />
      <EFProjectMetadata Include="TargetFrameworkMoniker: $(TargetFrameworkMoniker)" />
      <EFProjectMetadata Include="Nullable: $(Nullable)" />
      <EFProjectMetadata Include="TargetFramework: $(TargetFramework)" />
      <EFProjectMetadata Include="TargetPlatformIdentifier: $(TargetPlatformIdentifier)" />
    </ItemGroup>
    <WriteLinesToFile Condition=" '$(TargetFramework)' != '' "
                      File="$(EFProjectMetadataFile)"
                      Lines="@(EFProjectMetadata)" />
  </Target>
</Project>

and imported it in Startup.csproj with:

  <!-- To use dotnet ef without 'msbuildprojectextensionspath' -->
  <Import Condition="!Exists('$(BaseIntermediateOutputPath)\obj\$(MSBuildProjectFile).EntityFrameworkCore.targets')" Project="EntityFrameworkCore.targets" />

The strange part is that is that Startup.csproj.EntityFrameworkCore.targets exists in artifacts\obj\DbContext.

@ajcvickers, @roji, I think I have the correct solution for this. Just give me a few days.

[fardarter]

@paulomorgado How would this work with ArtifactsPath?

[paulomorgado]

@paulomorgado How would this work with ArtifactsPath?

@fardarter, can you elaborate more? I don't understand what you're asking.

[fardarter]

@paulomorgado How would this work with ArtifactsPath?

@fardarter, can you elaborate more? I don't understand what you're asking.

My current Directory.build.props has the following value:

<ArtifactsPath>$(MSBuildThisFileDirectory)/build/artifacts</ArtifactsPath>

That gives me a layout as per: https://learn.microsoft.com/en-us/dotnet/core/sdk/artifacts-output

So:

<root>/build/artifacts/obj/<projectname>/<configuration>

We have many projects referenced, but one startup project.

Will this scenario be covered?

And how would this be expressed in the current fixes?

[paulomorgado]

Querying properties from MSBuild execution will give you the exact values as during build.

You can test Evaluate items and properties and display results of targets for yourself and see the results.

What #34574 does is something like this:

dotnet msbuild -getproperty:AssemblyName,Language,OutputPath,Platform,PlatformTarget,ProjectAssetsFile,ProjectDir,RootNamespace,RuntimeFrameworkVersion,TargetFileNName,TargetFrameworkMoniker,Nullable,TargetFraemwork,TargetPlatformIdentifier,Platform  <csproj path>

[fardarter]

Querying properties from MSBuild execution will give you the exact values as during build.

You can test Evaluate items and properties and display results of targets for yourself and see the results.

What #34574 does is something like this:

dotnet msbuild -getproperty:AssemblyName,Language,OutputPath,Platform,PlatformTarget,ProjectAssetsFile,ProjectDir,RootNamespace,RuntimeFrameworkVersion,TargetFileNName,TargetFrameworkMoniker,Nullable,TargetFraemwork,TargetPlatformIdentifier,Platform  <csproj path>

I'm afraid I'm a bit out of my depth in translating that to a "yes, it will work" or "no it won't". Would you need to include the artifact path in the declaration here #23853 (comment)? Or is it derived?

[BladeWise]

I am currently using artifacts path, and adding a Directory.Build.targets (at the solution root) like this

<Project>
    <!-- To use dotnet ef without 'msbuildprojectextensionspath' -->
    <Import Condition="Exists('$(MSBuildProjectDirectory)\obj\$(MSBuildProjectFile).EntityFrameworkCore.targets')"
            Project="$(MSBuildProjectDirectory)\obj\$(MSBuildProjectFile).EntityFrameworkCore.targets"/>
</Project>

works as stated in #23853 (comment).

[paulomorgado]

At the moment, what I'm doing is roughly this:

1. Get the BaseIntermediateOutputPath from the project I want to run dotnet ef dbcontext script

dotnet msbuild -getproperty:BaseIntermediateOutputPath <path to target .csproj>

2. Run dotnet ef dbcontext script

dotnet ef dbcontext script --output <path to output> --startup-project <path to startup .csproj> --project <path to target .csproj>

And I don't have to make any changes to any project regardless of using artifacts output or not.