Can't process new ledgers: The current ETL source is not compatible with the version of the libxrpl Clio is currently using. Please upgrade Clio to a newer version (Version: 2.1.1~26ed78f-1)

[PaulMoney]

Issue Description

Hello, currently, I'm running my own small private rippled chain where I'm testing a new rippled plugin feature with a custom transaction type (e.g., TokenSwap). However, when I want to connect Clio to one of my custom rippled nodes, I receive the following error:

2024-05-15 10:44:00.555862 (etl/impl/AmendmentBlock.hpp:39) [0x00007f8c97fff700] ETL:FTL Can't process new ledgers: The current ETL source is not compatible with the version of the libxrpl Clio is currently using. Please upgrade Clio to a newer version.

I have upgraded Clio to the newest version available as Debian package (2.1.1~26ed78f-1), but the error persists. Also, what I have noticed, is that, if I manually specify start_sequence field in Clio configuration file, it starts extracting ledgers, however on some ledger were was TokenSwap transaction it stops and Clio reports that ledger with that index could not be found.

Is it possible to rebuild or include support for custom transaction types in Clio?

Steps to Reproduce

1. Build rippled node/nodes from this commit - https://github.com/mvadari/rippled/tree/d8e0cd1e6f51b6cdb91a99c4213afaaa35e968b8
2. Build xrpl-plugin, as noted here - https://github.com/mvadari/xrpl-plugin/tree/main/python
3. Send custom transaction (e.g., TokenSwap) to the rippled network.

Expected Result

Clio connects to custom rippled node and saves custom rippled transaction (e.g., TokenSwap).

Actual Result

Clio is unable to process new ledgers because current ETL source is not compatible with the version of the libxrpl Clio is currently using.

Environment

Ubuntu 22.04

[godexsoft]

Hi Paul,

Clio can't proceed with its ETL side of things if some unknown type arrives from rippled.
Is the custom type added in the plugin (in python) or in C++ code? I don't know enough about the plugin support so thought i'd ask.

In order to use a custom rippled (or really, libxrpl):

• In your custom rippled branch
  • Change the name in conanfile.py so it differs from the current one. I'd set it to something like xrpl-custom.
  • Run conan export ., this will export a local recipe for your custom libxrpl that introduces the new transaction type
• In your copy of Clio source code
  • Update conanfile.py to use the correct xrpl-custom package instead of the current xrpl one
  • Update deps/libxrpl.cmake by changing the required package to your xrpl-custom
  • Change xrpl:: to xrpl-custom:: in util/CMakeLists.txt
  • Now if you try to configure the project as per documentation, starting with conan install ... Conan will take your local library instead of fetching one from our artifactory

The workflow above is basically what is usually required when working on something that is not in libxrpl already.
There is an issue to improve the docs with the info above.

Please keep in mind that if you change the rippled code you need to conan export . again each time to avoid using the previous version and running into issues. This is the current downside of the workflow i described.

If your custom type is in python only, if that's even a possibility at all, then the above will not help. And frankly i don't see a way to even make it work. But hopefully new types can't be added outside of C++.

Hope this helps. Let us know if you need more help to get it to work 👍

[PaulMoney]

Hey @godexsoft Followed your steps in building custom Clio server, however, when I ran this command:

cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release ..

I received the following error -

CMake Error at src/util/CMakeLists.txt:27 (target_link_libraries):
  Target "clio_util" links to:

    xrpl::libxrpl

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

P.s. The first command -

conan install .. --output-folder . --build missing --settings build_type=Release -o tests=True -o lint=False

Works correctly without any issues.

Maybe I need to edit something else ?

[godexsoft]

Hi @PaulMoney, I think you also need to change xrpl:: to xrpl-custom:: (or what you used for the name in the previous steps) in util/CMakeLists.txt. As you updated the name of the package, now it should also be reflected where we try and link to the library. I'll update the steps above for completeness too.

[PaulMoney]

@godexsoft Now, the problem is with this command -

cmake --build . --parallel $(nproc)

When I run, I receive the following error -

In file included from /home/hero/.conan/data/xrpl-custom/2.2.0-rc1/_/_/package/c22f53cf03124b247fb1cf8a08a7c409b4b2716e/include/ripple/protocol/TxFormats.h:23,
                 from /home/hero/clio/src/util/TxUtils.cpp:22:
/home/hero/.conan/data/xrpl-custom/2.2.0-rc1/_/_/package/c22f53cf03124b247fb1cf8a08a7c409b4b2716e/include/ripple/protocol/KnownFormats.h:25:10: fatal error: ripple/plugin/plugin.h: No such file or directory
   25 | #include <ripple/plugin/plugin.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [src/util/CMakeFiles/clio_util.dir/build.make:328: src/util/CMakeFiles/clio_util.dir/TxUtils.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:274: src/util/CMakeFiles/clio_util.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

This error seems to be logical, as if I build rippled core from this branch - https://github.com/mvadari/rippled/tree/d8e0cd1e6f51b6cdb91a99c4213afaaa35e968b8

It has a new plugin folder, where plugin functionality is being placed.

Maybe you have any ideas how to workaround this ?

[godexsoft]

@PaulMoney I think this means that the plugin.h (and probably some others) are not actually in libxrpl but only in rippled itself. This would require to add them in the correct places in rippled's cmake files so these files are exported with the library.

Maybe @thejohnfreeman can give you some more concrete tips on this. I'm not too familiar with the build process of rippled.

[thejohnfreeman]

My suggestion: build both rippled and clio from source. The packages expect specific compatible versions as dependencies, but you want to use a custom version of rippled that is modified from an older version.

In your custom source tree for rippled:

• conan export the external recipes
• conan export the xrpl recipe. Note the full Conan reference (name/version@user/channel).

In a source tree for a version of Clio compatible with the base version of rippled that you modified:

• Change the xrpl requirement in conanfile.py to match exactly the reference you noted above. (You might be able to achieve this with the --require-override option to conan install in the next step.)
• conan install, cmake, cmake --build according to the build instructions for Clio.

[PaulMoney]

@thejohnfreeman Thanks for your response.

Could you please explain the exact commands I should run in source tree for rippled?

For now, I ran only this command - conan export . xrpl/2.2.0-rc1@hero/custom, and when I build Clio I receive the same error -

In file included from /home/hero/.conan/data/xrpl/2.2.0-rc1/hero/custom/package/5bb612f3ef45687cc5f5c326d3154007c5053ab8/include/ripple/protocol/TxFormats.h:23,
                 from /home/hero/clio/src/util/TxUtils.cpp:22:
/home/hero/.conan/data/xrpl/2.2.0-rc1/hero/custom/package/5bb612f3ef45687cc5f5c326d3154007c5053ab8/include/ripple/protocol/KnownFormats.h:25:10: fatal error: ripple/plugin/plugin.h: No such file or directory
   25 | #include <ripple/plugin/plugin.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [src/util/CMakeFiles/clio_util.dir/build.make:328: src/util/CMakeFiles/clio_util.dir/TxUtils.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:274: src/util/CMakeFiles/clio_util.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

[thejohnfreeman]

Ok I walked through the process and found two problems in the rippled branch you are using:

1. The Conan recipe is outdated.
2. The plugin public headers are not installed in Builds/CMake/RippledCore.cmake. (This is the kind of oversight that will become impossible once Finish physical redesign rippled#4997 is merged.)

I've fixed these problems in a branch in my fork. With that branch, the below script works:

# Need a build environment with Conan >= 1.60 and GCC >= 12. This is just one example.
sudo docker run --rm -it rippleci/clio_ci

# Check out the custom version of rippled you want to use.
git clone https://github.com/thejohnfreeman/rippled --branch plugins
pushd rippled

# These commands come from the build instructions for rippled.
# Conan is already configured in this container.
conan export external/snappy snappy/1.1.10@
conan export external/rocksdb rocksdb/6.29.5@
conan export external/soci soci/4.0.3@

# We don't have to build rippled yet. Just export it to local cache for Clio.
conan export . hero/custom
# Note the package reference you just exported.
reference="xrpl/2.2.0-b3@hero/custom"

popd
git clone https://github.com/XRPLF/clio.git
pushd clio

# Update the requirement to find the one we exported above.
sed -i "s|xrpl/2.2.0-rc3|${reference}|" conanfile.py

# These come from the build instructions for Clio.
mkdir .build
pushd .build
# There seems to be a dependency version conflict. Resolve it with `--require-override`.
conan install .. --output-folder . --build missing --settings build_type=Release -o tests=True -o lint=False --require-override zlib/1.2.13
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build . --parallel 8

[mvadari]

FYI I haven't done any work with Clio to make sure that it supports plugins; there's probably a fair amount of work on that front.

[PaulMoney]

Hey @thejohnfreeman
Thanks for your tries to build Clio.
I have followed your steps, however I received the following error when I ran the last build command:

[  9%] Building CXX object src/util/CMakeFiles/clio_util.dir/TxUtils.cpp.o
/home/hero/clio/src/util/TxUtils.cpp:22:10: fatal error: xrpl/protocol/TxFormats.h: No such file or directory
   22 | #include <xrpl/protocol/TxFormats.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [src/util/CMakeFiles/clio_util.dir/build.make:328: src/util/CMakeFiles/clio_util.dir/TxUtils.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:274: src/util/CMakeFiles/clio_util.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

[godexsoft]

@PaulMoney i had the same issue yesterday when updating my local env to latest Clio develop - delete the build folder and start over from the conan command. Seems like your build is still setup to use the previous libxrpl.

Or if you are on a very custom libxrpl/rippled branch then you want to merge latest upstream libxrpl into your branch because they recently changed paths (ripple -> xrpl)