Okay, so thank you very much for joining us for this talk where going to be hearing from John Evans
John is a member of the KeyCAD core development team and he is the author of the new KeyCAD
IPC interface that provides a scripting back end.
So he is going to give us an overview of what this IPC is and how to use it.
So without further ado, please welcome John Evans.
All right, thanks everyone.
So yeah, like Seth said, I've been working on KeyCAD for a long time.
If you're new to the world of open hardware and haven't heard of KeyCAD yet, welcome.
It's a CAD tool, mostly for designing electronics for printed circuit board design, but
it also has a few other applications.
So I'm going to be talking about one of the new features, which is coming in KeyCAD 9.
If you want to hear about the other 500 features, stick around for Wayne's talk a little
bit later, and KeyCAD 9 will be out shortly.
It was going to be out the end of this week, but it's going to be next week because
fuzz them and things so stick around.
So to talk about what I did here, I have to start by talking about why I did it.
If you're familiar with KeyCAD, you probably know that it already has a plug-in system.
KeyCAD has had the ability to add Python plug-ins for quite a long time.
And if you're really into KeyCAD, maybe you've even tried to write some of your own plug-ins.
So why did I make an API if we already have an API?
So the current Python plug-in system is mostly around a tool called Swig, which is a
clever tool that can analyze mostly C++ code.
Now they also support C, and it automatically generates wrappers in a variety of different
languages, and allows you to, for example, call C++ functions directly from Python.
So in the case of KeyCAD, a subset of our C++ application is wrapped using Swig into a Python
module, and plug-ins can then just import that Python module and call into KeyCAD code.
So this is pretty cool.
We don't have to do that much work.
Code is mostly generated for us.
We have some extra code that we had to write, but not that much.
And now Python-based tools can do a lot of different things with KeyCAD, which has been
really great for the ecosystem, because there is now a whole range of tooling that adds
on to KeyCAD does specific things or things that we hadn't yet supported in the main C++ code
base, which is pretty good, right?
Well, maybe this is a bit obvious in hindsight, but this really wasn't the best choice to deliver
an API that works the way that people expect when they hear the phrase API, like not having
the functions you call change every version, or not being able to crash KeyCAD, if you call
functions in the wrong order.
So the main challenge is with Swig, our cover, like a couple of different places.
So first of all, from the point of view of us, the KeyCAD development team, like I said,
the interface is really fragile.
There's no API layer within the KeyCAD code base.
It was just directly, let's take the existing things that KeyCAD is doing, let's make
it so that you can call them from Python.
And this means that we are constantly dealing with challenges when we want to refactor
that code, when we want to deprecate something, or add a new way of doing things, or just
change how we do things internally in order to improve KeyCAD itself.
We have to decide are we going to just break these plugins that are expecting this
old interface, which we sometimes have to do, unfortunately, or are we going to add some
way of preserving backwards compatibility, which sometimes is pretty easy and sometimes
is impossible, and often is forgotten about until later, which is annoying for all of the
Python developers.
In addition to that, which is a big pain for a lot of people, Swig is also a little
bit annoying for us on the development team, because we've been moving to use more and
more modern C++ APIs and types, and Swig is slowly getting better at this, but it's still
not, you know, it's not keeping pace with how we adopt C++ features.
And it also slows down the build process for us, which limits how fast we can iterate
on things.
And like I said, Python plugins are directly reaching into KeyCAD as it is running.
They're running in the same process, they have access to all that memory.
If they do things that don't match the expected contract, that we were thinking about
when we wrote the code, they can just completely break things in very weird ways and cause
an interesting bug reports.
But it doesn't stop there.
The current Python plugin system basically lets plugins assume that the Python bindings
to our core framework library, WSWJits, are available.
And sometimes this has been interesting for packaging, because that binding WX Python
is sometimes out of sync with WX widgets in different platforms, and we've had to patch
around that, and basically spend a bunch of energy maintaining things on the windows and
macOS side, especially.
We also have to bundle our own Python interpreter on those platforms, because we need to ensure
for these plugins that they have, all of these Python modules available with appropriate
versions, and we can't rely on that being installable externally.
So we have to now ship a Python environment, and plugins have to use that environment,
which is challenging for them if they want to then add external Python dependencies, and
sometimes it's just not possible for them to use certain Python dependencies.
There's one Python environment shared by everything, which if you're a Python developer,
you know, is just not a great idea.
If that wasn't enough, it's also not great as a developer if you're trying to build plugins,
because using keycads to run your Python code, the way that it runs is a little bit different
than the way that you would typically run a Python program, and so if you're trying
to test your code, if you're trying to debug in an IDE or anything like that, it's sometimes
possible, but it's very hard to get exactly the same environment and run it in an exactly
the same way.
So sometimes you just have to test things the brute force way.
We have a built-in Python console that lets you have a Python repel and just test things
out.
But if you're used to developing Python, maybe you're used to a little bit nicer tooling,
and this built-in console is something that the keycad team now has to maintain, and
it's another challenge and another thing that makes developing for keycad a little bit bespoke
compared to other kinds of Python development.
So that's where we are, and it's taken a long time to develop something better along the
way we considered a few options that we then didn't do, which I think are an interesting
dimension.
So first of all, we could have just made a real API layer in C++, and then pointed
to Swiget that instead of at the internals.
This would have been relatively straightforward, and it would have probably totally solved
those fragility problems where you know, function names are changing every keycad version.
The issue is that most of those other problems would have remained.
So we have known for a while that we wanted to move away from Swig.
And we also spend a little while looking at a library called Pi by 11, which is kind of serving
the same kind of function as Swig, but in a more manual way where you have more control
over how it works.
And this might have solved a few more of the problems, but definitely not all of them.
So about two years ago, I wrote an RFC for the team that basically said, what if we
went in a different direction here?
What if we could completely remove Python from the core of keycad and build a different
kind of API?
So over that window, I built a proof of concept, and it seemed to be working pretty well,
time passed, and here we are now about to ship the first real version of this with keycad
9.
So let's talk about the IPC API.
What I wanted in this API can basically be summed up like this.
First of all, we wanted to be able to change any internals in keycad and hide that change
from all the API users.
We should be able to do normal things that people expect, like have a sane deprecation policy
and have forwards and backwards compatibility wherever possible.
In the end of all, API users should no longer have direct access to keycad internal
state.
We want to run them in external processes and keep them from doing, let's say, accidental
evil.
And third, the developer experiences important, both for us the keycad team and for all
of the API users, if we want this thing to be adopted and useful, it has to be better
than what exists out there right now, not just different.
So here's how it works at a high level.
The API is using protocol buffers to define messages, messages that are passed back and
forth between keycad and an external application over unic sockets or Windows name pipes.
And we use a library called nanomessage next generation, which I'll just call NNG from
now on because it's shorter as an abstraction layer over those sockets and pipes.
So let's start at the bottom.
Why NNG?
It's a question I get sometimes.
I looked at a number of different options that could solve the problem I had where I want
to send bytes back and forth between keycad and other things on the same machine.
And basically, NNG came out on top because it's pretty simple to use and very few lines
of code.
And those very few lines of code are using the native platform IPC mechanisms.
The same code does the right thing on Windows or on non-Windows and you don't have to
care that much about the differences there.
And it has all the availability we needed it to be in terms of what platforms it supported
and what languages it supported.
And basically, I didn't find a good reason not to use it.
And I also looked at a lot of other options so you can see some of them down here.
And all of these other options came out as either more complicated or they had some drawback
or in some way they were trying to solve a different set of problems than my problems.
So I think there are great solutions for different problems but they weren't the one for me.
So then one layer up, yProtobuff.
So if you're not familiar with Protobuff, basically, is a project from Google where you define
messages using a domain-specific language.
And then use a tool that compiles that DSL into various different wrappers essentially
in a bunch of different target languages that implement those messages.
And we use this because we can then define most of the keycad API in terms of those DSL
messages which are pretty easy to parse but look at some in a bit.
And it means it's really easy to keep keycad synced up with the third party language bindings.
So Protobuff is not perfect and in fact it has some downsides that are important to be aware
of if you want to use it for your project.
But ultimately I did look at a number of alternatives to Protobuff and they either didn't
have all the features we need or they weren't nearly as popular.
And I decided after looking at what was available and all of the resources I could find,
people posting about challenges they had had and how they had overcome those challenges.
I wanted to go with the thing that was the most widely used because I could see how these
people were doing similar things and how they had solved these problems.
And some of the alternatives I looked at, there just wasn't that much information about people
solving similar problems.
And I think that kind of thing is important.
So here are some example Protobuff messages from the Kika API.
You can think of them kind of like structures if you know a language like CRC++.
You can see some of these have primitive types like these in 64s and then a lot of them
are composed out of other messages internally.
So here's an example of a command that an API client might want to send to get the bounding
box or extends from a text object.
So we're going to have this certain other message that specifies what the text is and
we're going to get back that box 2 we just saw with the bounding box.
So to actually send and receive these messages, we have these what are called envelope
messages that have a request and response inside as an any type which is a Protobuff feature
that basically lets you do runtime type introspection so that we can dynamically dispatch a handler
depending on what kind of message we receive coming in.
Protobuff defines a binary format that these messages get serialized into and that serialized
format is what we send back and forth over the wire.
And one note is that Protobuff can also be serialized as text that is basically JSON.
So we have a debugging feature where you can actually dump every message going back and
forth to a text log and see what's going on which is really handy.
So we have these messages, how do we actually hook them up to Kikaad?
At a super high level, Kikaad is an event driven application.
One of our other devs, Ian, who's up there, did a deep dive into this on our developer
meetup on Friday and you can watch that on our YouTube channel if you want the full details.
But at a really high level, events are coming in from various sources, maybe from an input
device, maybe from the operating system and they get routed to various event handlers
by it at Stasher.
This is all happening in a single thread.
So all these handlers are designed to be non-blocking and sometimes they have their own
internal event loops and all of this is going on with various core routine magic that
lets us have somewhat sane code, but I'm going to skip over that today.
So to hook into this, I basically added a new type of event and this is called whenever
any kind of API message comes in and that is routed to a new kind of handler which is
handling the API messages.
And this handler is unpacking that envelope message and figuring out, okay, what should I
do with this?
It has its own set of handlers that are all supporting the actual API functions.
Some of those functions might be able to return some calculation immediately.
Some may be able to create their own other events and post those to the event loop that
will then get handle that some later time.
So all of these have to be non-blocking and essentially return to the client immediately.
So if you want to call an API command that is going to take some time to do, you are
just going to immediately get a reply that, hey, this command started and you are going
to need to check back to see when it finished.
Some of the details of this by the way can be kind of hidden from the actual users of the
API through the client libraries to make it a little bit less complicated.
And one other thing I should mention is that these API handlers are dynamically registered.
So for example, if you open a keypad project manager, you might only have a few handlers,
but then when you open up the PCB editor, it will register a PCB handler because now you
can work with a board and now you can call get board and refill zones and things like that.
And if you close that PCB editor, it will get the registered and then if you say refill zones,
you will instead get back a message saying, hey, there's not a handler for that right now.
So because of this design, the API events are synchronous and are getting sequenced in with
all of the user input, so that's something you have to keep in mind as a plug-in developer
is that you're not kind of operating in parallel with the user, but with the user.
So you have to keep in mind how you want your plug-in actions to get sequenced in as under
steps.
And sometimes the user will be doing something that we don't want to allow interruption,
like if you're in the middle of routing tracks, you're API plug-in to be able to just delete
the board.
So sometimes the API will say, hey, I'm busy, you need to try that again later.
So that's an overview of how it's implemented, but what about using it?
Well, we're releasing Python bindings for the API along with HIKI 9, and you can actually
install it right now if you have a nightly and want to try it out.
So it's there in the Python package index, and that's the source code for it.
Just make sure that you go into your KIKI preferences and turn on the API because it's not
on by default right now.
So here's a simple example of creating a zone on whatever board is open.
We start by importing some things, and then getting a handle to the open KIKI instance,
and then the board, and then we construct a zone object that has certain properties, which
is most of the code you see there, and finally, we put it on the board.
By the way, we could be developing this from any IDE or from a REPL outside of KIKI,
and it would work the same way as if it was running as a plug-in from KIKI.
So there's no longer any difference there.
You can use whatever kind of Python environment is most comfortable for you.
One thing I want to point out is that these two lines are the only lines that are communicating
with KIKI using the API.
So the rest of it is building up objects and Python that are representing protobuff messages
that will then be sent.
You could also take away that hunt.
It's a lot of steps to define an outline for a zone.
And this is true.
And I guess my point here is that even if we make the API really easy to use and Python
and all that good stuff, you're still going to need to kind of understand KIKI some places
to understand, okay, the zone is complicated because it can actually have a arcs in its border
and it can have internal cutouts and things like this.
And so the API describes all of that.
So that just is on us to make sure that all of these things are documented well
and have good examples for you to work from.
One thing that I'm not going to have time for too much detail on is that there is a new
Python plugin launching system coming with KIKI 9, which is using virtual environments
per plugin and automatic dependency installation that mostly works.
And this, you know, I'm sure there's a few edge cases because that's Python where it
doesn't work, but for a few people who've been testing it, it seems to work, which is
really awesome because now plugins can use whatever UI to locate they want.
And as a bonus, it can launch any executable file so you don't necessarily need to use
Python if you don't want to.
So KIKI 9 is coming from with the first public version of this API, but I'm definitely
not done.
There are a lot of features that are important to add that will be coming in future versions.
And probably some things I don't even know I should add.
So I'm definitely looking for feedback from early users of this, you know, up next I'm
going to be working on integration with the footprint editor and replacing the footprint
wizard systems so that we can have a shared code base between our library team and KIKI
had itself, you know, called back to the CAD query talk, which just happened.
We also want to support headless operations.
And since I know a lot of people are wondering, yes, we are bringing this to schematic
and symbols.
No, it's not in nine.
Basically, we just ran out of time to do the necessary internal changes to the schematic
editor to make that possible.
So if you want to learn more, you can find a little bit more detail than I was able to cover
today in the DevDox link here as well as more information about creating new API plugins.
KIKI Python is up on Python package index so you can check that out and there are some
examples, basic plugins and also just some quick scripts in that repo.
So with that, any questions?
Hi, I was just wondering, how does that work when you have multiple KIKI at instance running,
like, how do you select one or something like that?
Great question.
Question was, if you have multiple different instances, how do you know which one to talk to?
So there will be a default path that the first instance will grab.
And then after that, if it's already taken, you will have a different way to communicate
that will have different process ID in the path essentially.
When you launch a plugin from KIKI, it passes the information through environment variables
as to where you should communicate with this KIKI that just launched it.
And if you are running from a debugger, you would have to manually override that if you didn't
want to use the default path.
So if you're developing a Python plugin, I recommend using just one instance at a time.
Can you go a bit into, sorry, can you go a bit into how the virtual environment installation
works on, like, how much Python installation?
Yes, so the virtual environment is today pretty basic and assumes one particular workflow,
which has so far been working in my tests, but I know it won't cover all cases.
So basically, if you're plugin in your plugin definition file, says it's a Python plugin,
then we will look for a requirements text file, and we will attempt to install.
So we will use Python, basically you can configure which Python interpreter to use,
and then we will assume that we have access to the virtual environment module.
So if you somehow don't have that, it won't work.
And then we will attempt to install PIP, and then we will attempt to use PIP to install your dependencies.
And if you can't install PIP, or if your dependencies don't have binary wheels, it won't work,
and you'll have to do something manual.
But that's something that I'm sure could be improved in the future.
Well, we have more questions if our pod, if you could start getting set up.
One question if my plugin is doing something both with schematics and PCB editors,
are a way that I can launch any application.
You mentioned that it has to be running in order to understand the command.
So if you want to have a plugin that launches kick-out?
No, if my plugin is like printing schematics and generating Gorbors, for example,
it has to deal with schematics and PCB new.
And if one of them is not running, one will happen.
You mentioned that schematics has to be running in order to understand commands for schematics.
Yeah, so today you can only with the current plugin system.
It's intended for the use case of the user launches it from the editor.
So your plugin cannot open the editor.
The user has to open the editor and then launch the plugin.
In the future, we won't have that restriction.
One of the things I put on the roadmap was a headless mode.
So right now you actually can't do plotting and exporting with this.
That's well supported by keycud CLI, so that's our recommendation right now.
But in the future, you will be able to do things like that through a special mode of keycud CLI,
where you can have your plugin say, now I want to open this schematic file and work with it.
Any further questions?
Okay, thank you, John.