Today in Edworking News we want to talk about Crossing the Impossible FFI Boundary, and My Gradual Descent Into Madness
Anyone trying to make a new mainstream language is completely insane, unless they're backed by a huge corporation. There are only two exceptions in the last 25 years that come close: Scala and Kotlin. They did this by seamlessly building on an existing ecosystem, specifically Java's. But what if you’re a low-level, memory-safe language like Vale, Austral, or Ante? There's no existing ecosystem of fast, memory-safe code to use. And unfortunately, seamlessly building on Rust's ecosystem is impossible... ...or so we thought!
The Impossible Task
Anyone who has tried to make Java call C, or Python call Javascript, or C call Rust, can tell you that it's really difficult to call functions from other languages. Heroes throughout the ages have created tools like SWIG, CXX, etc., which at least make it possible for a user to reach across the Foreign Function Interface (FFI) boundary, if they can figure out the correct incantations. But even with these tools, it's so difficult that we often just give up and call a microservice instead.
Calling into Rust is even harder; you can't just slap an `extern` onto your function, because Rust doesn't have a stable ABI; rustc-compiled functions don't have a predictable signature that the linker can recognize. Moreover, we can't send normal Rust objects to C. We instead have to make new structs with `#[repr(C)]`, which have restrictive rules about what they can contain. There are even more challenges here, making it an almost uncrossable chasm.
Overcoming Rust Challenges
Our ultimate goal is to write this Vale code and successfully make a Rust Vec and call capacity on it. The proof-of-concept tool demonstrates how C can treat Rust objects as opaque types, basically raw blobs of bytes that cannot be read directly.
Under the hood, the tool automatically generates some Rust code using `cbindgen` to generate an equivalent C header into "rust_deps/rust_deps.h". However, it mysteriously left out the hardest part: How does the tool get the correct information from Rust to generate the required Rust code?
Gathering Type Information
The tool gathers necessary information by running a small Rust program and reading its output. This program uncovers three pieces of information about `std::vec::Vec`: its size, alignment, and field types. Although this method is somewhat weird, it is straightforward. To gather function information, the tool attempts to generate the Rust code corresponding to a given function and its properties.
A Horrifying Idea
Initially, various strategies were considered, such as using macros or the syn crate, but these did not have the required information. A thought turned to rustc's MIR (Mid-level Intermediate Representation). Fortunately, experts suggested using rustdoc's JSON output instead, which was a much better solution. The tool invokes rustdoc and reads the resulting JSON using `rustdoc_types`, organizing various relationships in HashMaps.
Implementing Overload Resolution and Generics
When encountering overloaded functions, the tool examines all implementations for a given Rust crate and identifies which ones match user specifications. This was a complex task due to Rust's generic types and varied overloads. After exploring various ideas, a better approach was discovered by leveraging rustdoc's JSON outputs and refining the tool's methodologies.
Ultimately, the Solution
Eventually, by persisting and experimenting, the team devised a surprisingly succinct and effective solution for resolving function overloads. The new Rust-based solution replaced overly complex code, making it simpler without sacrificing functionality. Removing thousands of lines of redundant code felt incredibly liberating.
Insights and Future Directions
The final approach simplifies crossing the FFI boundary between Rust and C, opening doors for seamless integration. The tool's potential in code generation and other applications is immense, encouraging future innovations and explorations.
Remember these 3 key ideas for your startup:
- Innovation Through Persistence:
Success often requires innovative solutions and persistent efforts. Overcoming technical challenges, no matter how daunting, is crucial for startups working with new technologies. - Leveraging Community Knowledge:
Engage with the community and experts to find better approaches and solutions. Collaboration can reveal simpler, more effective methods than those initially considered. - Simultaneous Simplicity and Complexity:
Strive for solutions that balance simplicity and functionality. Eliminating unnecessary complexity can lead to more streamlined and robust implementations.
Edworking is the best and smartest decision for SMEs and startups to be more productive. Edworking is a FREE superapp of productivity that includes all you need for work powered by AI in the same superapp, connecting Task Management, Docs, Chat, Videocall, and File Management. Save money today by not paying for Slack, Trello, Dropbox, Zoom, and Notion.
For more details, see the original source.
Onward to new horizons!
