I might need a little help interpreting how to fix the test that failed in CI too

There's also the possibility of opening it up to include repos that have .podspec or Cartfiles

I for one would welcome that a lot, since there's many packages out there doing their best to support all package managers so as to let users use whatever they already have included in their project. I realize that many packages also eagerly added a package manifest in 2014 without ever maintaining it, especially those packages where it just didn't make sense (iOS only packages for example), but that's changed now/soon with the release of SwiftPM support in Xcode 11.

I might need a little help interpreting how to fix the test that failed in CI too

There's a missing newline at the end.

Hmm added a new line but that didn't seem to do it

Better than editing, launch the jq command to sort
I take your fork in my fork, launch the command, push and my travis is ok

Thanks for all new projects

For the PR, I know URL do not care about case, but it easier to read for human

This project SwiftPMLibrary have case on GitHub.
Less work to do for dev, just copy it from Github which propose it with case, etc..

So I am against changing to the case.
(if adding a new project is not an automatic process, or a submission form or command line like cocoapods)

Fixed the lowercasing problem so all existing entries are undisturbed and ranjq locally. Looks like everything passes.

Up for discussion is whether my crawler could be of any use now or in the future. If i rerun it from the beginning and changed rules to include podspec and cartfile packages and omit packages < 4.0 (i guess by checking the swift-tools-version?), it wouldn't surprise me if it took a week to run, given the API rate limitations. I think i get about 500 calls per hour and each package analysis takes 3-4 calls each.

For reference this is the crawler code, currently dependent on Cloudant because I used IBM Serverless to run it: https://github.com/bitwit/swift-package-directory/blob/master/Sources/SPDCore/Services/PackageCrawler.swift

This is fantastic, thank you Kyle!

I think some of the reason that there isn't a lot of overlap is that my original crawl had a lot of these packages in, but I removed them when they were causing my parser to fail. You can see some of that work here.

A couple of examples, just from loading a few random repositories here.

• https://github.com/a2/passcards-swift - Has a Package.swift but doesn't declare any libraries or executables.
• https://github.com/aktowns/csdl - Not really a Swift package.

That said, my has got much more reliable since then and so I think with a couple of simple changes this could be good to merge because the vast majority of packages added here are fantastic, and completely legitimate to be in the directory.

I'm going to hold off on merging just for now though. I'm in the middle of a huge change to the back end code and I have no doubt that when I add these packages it'll find plenty of edge cases in my parser. I'd rather be working with a stable codebase when that happens.

I should merge my big change in either today or tomorrow, so I'll look at this again towards the end of the week.

Just one more thing on this, I think I'll include a new rule in the search that only packages that define at least one library or executable should be eligible for searching. I have this data already so it will be easy to do. Tracking this as #106.

https://github.com/aktowns/csdl - Not really a Swift package.

Isn't it though? That specific package uses an old manifest format, but it's still a Swift package nonetheless that others might want to include as a dependency.

Isn't it though? That specific package uses an old manifest format, but it's still a Swift package nonetheless that others might want to include as a dependency.

You're right, of course, but the majority of packages in this state would include Swift code, and that Swift code would be so out of date that its usefulness would be questionable.

The other issue with this package, and many others, is that they won't process using dump-package with the Swift 5/5.1 tools.

I currently get the package name from that generated JSON manifest. Of course, I could keep a copy of Swift 3 around for these packages, and then fall back to trying that when 5.x fails, but I'm just not sure it's worth the work to support these packages that are multiple years old.

Oh if it's about the old manifest format, I'm totally with you. If a package fails to be to analyzed with current tooling it shouldn't be included (or not updated if it's already being indexed). I thought this was about the package only being a modulemap. If dump-package doesn't work on that then that's probably a bug worth filing a JIRA for.

I thought this was about the package only being a modulemap.

It kinda was, but you convinced me on that one straight away. I think this package is a bit of a special case where it hasn't been updated in 4 years, but is still potentially relevant and useful as it includes no actual Swift code. An edge case for sure, but a valid one. 👍

If dump-package doesn't work on that then that's probably a bug worth filing a JIRA for.

This is the practical side of it though, running dump-package gives:

$ swift package dump-package 
/Users/dave/Desktop/CSDL: error: package at '/Users/dave/Desktop/CSDL' is using Swift tools version 3.1.0 which is no longer supported; consider using '// swift-tools-version:5.1' to specify the current tools version

Apologies for the very long delay on getting this merged, I've had a few other things that needed attention first. I just merged it and the server is starting to chew through the packages now… Fingers crossed!

Thank you so much for your efforts on this @bitwit, I really appreciate them.

As expected, this uncovered all sorts of edge cases in my code 😂 The final run through these packages just finished though. It's now tracking 3429 packages with over 42,000 package versions as of right now.

This is great news. Super psyched to have been able to contribute to this project and extract more value out of past experiments. If there's anything I can do with the crawler code to assist you in the future, please let me know.