Gentle Up⭐️Star — Gentle Up the Street to Open Supply!

Try GitHub: https://github.com/apache/incubator-seatunnel

Throughout the latest stay occasion of the SeaTunnel Connector Entry Plan, Beluga open supply engineer Wang Hailin shared the “SeaTunnel Connector Entry Plan and Growth Information to Avoiding Pit,” and taught everybody how one can develop a connector from scratch, together with the entire course of – from preparation to testing, and closing PR.

Wang Hailin, Beluga Open Supply senior engineer

Wailin Hailin is an open supply fanatic, SkyWalking Committer, and DolphinScheduler and SeaTunnel contributor. His present work focuses on efficiency monitoring, information processing, and extra. He likes to review associated technical implementations and take part in group exchanges and contributions.

This presentation is split into 5 elements:

1. Concerning the connector entry incentive program

2. Preparation earlier than claiming/creating connector

3. Small issues in growth

4. Issues for writing E2E Assessments

5. Preparations to submit a PR

1. Concerning the Connector Entry Incentive Plan

Firstly, let me introduce the SeaTunnel Connector Entry Incentive Program, and the steps to develop a connector from begin to end (even for novices). This consists of the entire strategy of preparation for growth, testing, and closing PR.

The SeaTunnel group launched a brand new connector API not way back, which helps operating on varied engines, together with Flink, Spark, and extra. This eliminates the necessity for repeated growth of the outdated model.

After the brand new API is launched, the outdated connector must be migrated, or the brand new connector needs to be supported.

With the intention to inspire the group to actively take part within the SeaTunnel Connector Entry work and assist construct SeaTunnel right into a extra environment friendly information integration platform, the SeaTunnel group initiated actions, sponsored by Beluga Open Supply.

The actions have three modes: easy, medium, and arduous for the duty of accessing the connector. The brink is low.

You possibly can see which duties must be claimed on the exercise problem checklist, in addition to segmentation primarily based on issue and precedence. You possibly can select the exercise you might be snug with. You can begin contributing primarily based on the issue degree.

The ecological building of SeaTunnel can develop into extra full and superior solely with the assistance of your contributions. You’re welcome to take part actively.

With the intention to categorical our gratitude, our occasion has arrange a hyperlink the place factors will be exchanged for bodily prizes. The extra factors you get, the extra prized you may win!

Presently, we’ve seen many small companions take part within the occasion and submit their connectors. It’s not too late to hitch as there’s nonetheless a big time period earlier than the occasion ends. Primarily based on the issue of the exercise, the deadline could also be relaxed or prolonged.

2. Preparations Earlier than Claiming/Creating Connectors

So, how do you become involved with this wonderful exercise?

By first attending to know the fundamentals of a connector.

01. What’s a connector?

A connector consists of Supply and SInk (Supply + Sink).

Within the above determine, the connectors are linked to numerous information sources on the higher and decrease layers. The supply is liable for studying information from exterior information sources, whereas the sink is liable for writing information to exterior sources.

There may be additionally an abstraction layer between the supply and the sink.

By way of this abstraction later, the info varieties of varied information sources will be uniformly transformed into the info format of SeaTunnelRow. This enables customers to arbitrarily assemble varied sources and sinks, in order to comprehend the combination of heterogeneous information sources, and information synchronization between a number of information sources.

02. The right way to declare a connector

After understanding the essential ideas, the subsequent step is to say the connector.

GitHub hyperlink: https://github.com/apache/incubator-seatunnel/points/1946

You should utilize the above-mentioned GitHub hyperlink to see our plans for connecting to the connector. You can also make any additions at any time.

First, discover a connector that has not been claimed. To keep away from conflicts, search the whole problem to see if anybody has submitted a PR.

After claiming the connector, we propose that you just create a problem of the corresponding characteristic, synchronize the issues you encountered within the growth, and talk about the design of your answer.

If you happen to encounter any issues and need assistance, you may describe them within the problem, and the group can take it up collectively. Take part within the discussions to assist clear up the issue. That is additionally added to the document of the operate implementation course of, which makes it straightforward to consult with when sustaining and modifying sooner or later.

03. Compile the mission

After claiming the connector, it’s time to organize the event atmosphere.

First, fork the SeaTunnel mission to the native growth atmosphere and compile it.

Right here’s the compilation reference documentation: https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/setup.md

Run the testcase within the documentation after the compilation is profitable. You may encounter some points/issues throughout the first contact compilation course of, akin to the next compilation errors:

The answer to the above exceptions:

1. rm {your_maven_dir}/repository/org/apache/seatunnel

2. ./mvnw clear

3. Recompile it

04. Perceive Connector associated code construction

The success of mission compilation signifies that the event atmosphere is prepared. Subsequent, let’s check out the mission code construction and API interface construction of the connector.

Engineering Code construction

After the mission is compiled, there are three elements associated to the connector. The primary half is the code implementation and dependency administration of the brand new connector module.

The second half is the instance. When testing domestically, you may construct a corresponding case on the instance to check the connector.

The third half is the E2E-testcase: including focused take a look at instances on the respective operating engines of Spark or Flink, and verifying the useful logic of the connector by automated testing.

Code construction (interfaces, base courses)

The general public interfaces and base courses used within the growth are absolutely described in our readme. For instance, API operate utilization eventualities.

Right here’s the hyperlink: https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/README.en.md

05. See how different folks develop connectors

After going by the above steps, don’t rush to start out the work. As an alternative, first, try how others do it.

We strongly suggest you try the connector novice growth tutorial shared on the group official account:

As well as, you may consult with the merged Connector code to see the scope of adjustments, the general public interfaces and dependencies used, and the take a look at instances.

https://github.com/apache/incubator-seatunnel/pulls?q=is:pr+is:merged+label:connectors-v2

3. Small Points/Activity Throughout Growth

Subsequent, it’s a must to formally enter the connector growth course of. What issues could also be encountered throughout the growth course of?

The connector is split into the supply and sink ends — you may select both one or each.

01. Supply-related growth

The very first thing to concentrate to when creating a supply is to find out the studying mode of the supply: is it streaming or batch? Is help nonetheless required?

Use the Supply#getBoundedness interface to mark the modes supported by the supply.

For instance, Kafka naturally helps streaming studying, however it might additionally help batch mode studying by acquiring lastOffset within the supply.

One other query to concentrate on: does the supply require concurrent reads? Whether it is single concurrency, after the supply is began, a reader will likely be created to learn the info from the info supply.

If you wish to obtain multi-concurrency, you want to implement an enumerator interface by which information blocks are allotted to readers, and the readers every learn their allotted information blocks.

For instance, the Kafka supply makes use of partition sharding, and the jdbc supply makes use of fields for vary question sharding. It needs to be famous right here that if it’s a concurrent studying methodology, the steadiness of the info block distribution guidelines have to be ensured.

It is because at the moment, the connector has a corresponding enumerator on every shard in precise operation, and it’s mandatory to make sure that the enumerator has information in every shard.

Thirdly, does the supply must help resumable switch/state restoration?

If you wish to help this, you want to implement:

• Supply#restoreEnumerator: restore state

• Enumerator#snapshotState: storage shard allocation

• Reader#snapshotState: shops the learn place

02. Sink-related growth

If the sink is a typical sink implementation, use Sink#createWriter to jot down our information in line with the concurrency of the supply.

If you want to help failure restoration, you want to implement:

If you wish to help two-phase commit, you want to implement the next interfaces:

03. Connector associated

Subsequent, let’s check out among the common issues, particularly when the primary contribution is made with completely different types for every atmosphere, there are sometimes varied issues. Subsequently, it’s endorsed that you just import instruments/checkstyle/checkStyle.xml from the mission throughout growth, and use a unified coding format.

Whether or not it’s a supply or a sink, it is going to contain defining the info format. The group is pushing for a unified information format definition.

If you happen to really feel that the compilation velocity is sluggish, you may quickly annotate the outdated model of the connector-related module with a view to velocity up each growth and debugging.

04. The right way to search assist

While you encounter issues throughout growth and need assistance, you may:

• Describe the issue in your Difficulty and name lively contributors

• Focus on on mailing lists and Slack

• Talk by the WeChat group (when you have not joined, please observe the SeaTunnel official account to hitch the group, and add a small assistant WeChat seatunnel1)

• There could also be a group docking group for docking third-party parts (permitting you to do extra with much less).

4. Notes on Writing E2E Assessments

E2E testing is essential. It’s typically referred to as the gatekeeper of connector high quality.

It is because, if the connector you wrote will not be examined, it might be troublesome for the group to guage whether or not there are issues with the implementation of the static code.

Subsequently, E2E testing will not be solely useful verification but in addition a strategy of checking information logic, which may scale back the stress on the group to assessment code and guarantee fundamental useful correctness.

In E2E testing, these are among the issues which may be encountered:

01. E2E Failed – Check Case Community Handle Battle

As a result of the E2E community deployment construction has the next traits:

• Exterior parts that Spark, Flink, and e2e-testcase rely on (for instance, MySQL), use the container networkAliases(host) because the entry deal with

• e2e-testcase on either side of Spark and Flink might run in parallel below the identical host

• Exterior parts that e2e-testcase is determined by, must map ports to hosts for e2e-testcase to entry

Subsequently, E2E has to concentrate to:

• The exterior parts e2e-testcase is determined by the ports mapped to the exterior networkAliases, and so can’t be the identical within the testcases on either side of Spark and Flink

• e2e-testcase makes use of localhost, the above-mapped port, to entry exterior parts

• e2e’s configuration file makes use of networkAliases(host), the exterior parts that rely on port entry within the container

Right here’s the E2E Testcase reference PR: https://github.com/apache/incubator-seatunnel/pull/2429

02. E2E failure – Spark jar bundle battle

Spark makes use of the father or mother first-class loader by default, which can battle with the bundle referenced by the connector. For this, the userClassPathFirst classloader will be configured within the Connector atmosphere.

Nonetheless, the present packaging construction of SeaTunnel will trigger userClassPathFirst to not work correctly, so we created a problem, https://github.com/apache/incubator-seatunnel/pull/2474, to trace this problem. Everyone seems to be welcome to contribute options.

Presently, this may solely be resolved by changing conflicting packages within the spark jars listing with the documentation.

03. E2E failure – Connector jar bundle battle

Each the outdated and new variations of Connector are depending on the E2E mission and trigger conflicts.

PR https://github.com/apache/incubator-seatunnel/pull/2414 has resolved this problem.

Model battle between Connector-v2:

• Primarily happens throughout E2E, as a result of the E2E mission is determined by all Connectors

• We might plan to offer a separate take a look at mission for every Connector (or model) sooner or later

04. Inadequate E2E – Sink Logic Verification

The FakeSource of the Connector-v2 model can solely generate random information of some mounted columns at current, and the group companions are optimizing it to make it higher. https://github.com/apache/incubator-seatunnel/pull/2406

That mentioned, we are able to quickly clear up this downside by simulating the info of the required content material by Rework#sql:

05. Inadequate E2E – Supply validation information

The Assert Sink can configure column guidelines, however can not do row-level worth checking. For this downside, you may quickly use different connector sinks with exterior storage for question verification information.

06. E2E stability enchancment

In lots of instances, when E2E begins, you may use Thread.sleep to attend for useful resource initialization. Right here, sleep will trigger fewer initialization failures however extra time-wasting points.

As well as, as a result of instability of assets, community, and different points, you may be capable to run it now however not later.

To keep away from this downside, Thread.sleep will be changed with Awaitility.

07. A way to hurry up E2E

At current, I see that most individuals run E2E exams individually for each supply and sink. If you wish to velocity up the PR course of, it’s endorsed that you just mix each the sink and supply into one E2E testcase for verification, and run the testcase solely as soon as.

5. Checks Earlier than Submitting a PR

After finishing the earlier steps, please be sure you do some checks earlier than submitting PR – together with the next features:

Full recompile mission:

1. Codestyle validation, dependency validation

2. The profitable compilation earlier than doesn’t imply that it may be compiled efficiently now

Working E2E domestically succeeds:

1. Each Flink and Spark are verified

Complement or change the doc and assessment it once more earlier than submitting:

1. Assessment for locations not coated by exams

2. Locations that hav been reviewed earlier than and must be checked once more

3. Assessment for together with all recordsdata, not simply code

The above operations and steps can enormously save CI assets, velocity up PR Merged, and scale back the prices of group critiques.

Apache SeaTunnel

– Keep up a correspondence –

Come and Develop With the Group!

Apache SeaTunnel (incubating) is a distributed, high-performance, simply scalable, information integration platform for large information (offline & real-time) synchronization and transformation.

We’re more than pleased to welcome extra folks to hitch!

Although the journey of SeaTunnel (previously Waterdrop) has simply begun after having the ability to enter the Apache Incubator, the event and development of the group require that extra folks be a part of.

We imagine that below the steerage of The Apache Means, akin to “Group Over Code,” “Open and Cooperation,” “Meritocracy” (meritocracy), and “Variety and Consensus Resolution Making,” we are going to usher in additional numerous and inclusive group ecology, and collectively construct the technological progress introduced by the spirit of open supply!

We sincerely invite all companions who’re inquisitive about making native open supply to hitch the SeaTunnel contributor household and construct open supply collectively!