Connectors with InfinyOn Cloud is the best option for those who want to manage their data pipelines in one place.
Configuring connectors works the same way with InfinyOn Cloud, as it does locally.
You can create and maintain your connectors in a more streamlined way through the CLI with fluvio cloud.
This guide will walk you through creating an Inbound HTTP connector to ingest json data from and HTTP endpoint.
To follow this guide you will need to sign up for InfinyOn Cloud and log into the CLI.
fluvio cloud login
This is the config file for the Inbound HTTP connector in this guide.
# catfacts-basic-connector.yml
version: 0.4.3
name: cat-facts
type: http-source
topic: cat-facts-data
parameters:
endpoint: https://catfact.ninja/fact
interval: 30s
In this config, we are creating a connector named cat-facts. It will request data from a cat fact API once every 30 seconds and receive json data. The connector will store the json into a topic called cat-facts-data
You can create a connector by using fluvio cloud connector create with the example connector.
$ fluvio cloud connector create --config catfacts-basic-connector.yml
connector "cat-facts" (http-source) created
After the connector is created, you can list the connectors you’ve created, and view their current status.
$ fluvio cloud connector list
NAME TYPE VERSION STATUS
cat-facts http-source 0.3.0 Running
If there is a need to debug the behavior of a connector, the logs are available by running fluvio cloud connector logs cat-facts
$ fluvio cloud connector logs cat-facts
2022-10-23T07:36:36.915928Z INFO http_source: Starting HTTP source connector connector_version="0.3.0" git_hash="10ee08a94b7be7d91a31a01104b7f6e86e54b7d9"
2022-10-23T07:36:36.915981Z INFO http_source: interval=30s method=GET topic=cat-facts output_parts=body output_type=text endpoint=https://catfact.ninja/fact
2022-10-23T07:36:36.916165Z INFO fluvio::config::tls: Using verified TLS with certificates from paths domain="odd-butterfly-0dea7a035980a4679d0704f654e1a14e.c"
2022-10-23T07:36:36.920362Z INFO fluvio::fluvio: Connecting to Fluvio cluster fluvio_crate_version="0.12.14" fluvio_git_hash=""
2022-10-23T07:36:36.979270Z INFO connect: fluvio::sockets: connect to socket add=fluvio-sc-public:9003
2022-10-23T07:36:37.025300Z INFO connect:connect_with_config: fluvio::config::tls: Using verified TLS with certificates from paths domain="odd-butterfly-0dea7a035980a4679d0704f654e1a14e.c"
2022-10-23T07:36:37.088073Z INFO connect:connect_with_config:connect: fluvio::sockets: connect to socket add=fluvio-sc-public:9003
2022-10-23T07:36:37.494092Z INFO dispatcher_loop{self=MultiplexDisp(13)}: fluvio_socket::multiplexing: multiplexer terminated
2022-10-23T07:36:37.544828Z INFO http_source: Connected to Fluvio
2022-10-23T07:36:38.060832Z INFO run:create_serial_socket_from_leader{leader_id=0}:connect_to_leader{leader=0}:connect: fluvio::sockets: connect to socket add=fluvio-spu-main-0.acct-.svc.cluster.local:9005
The HTTP connector should be receiving data and storing it in a topic with the name we specified.
$ fluvio topic list
NAME TYPE PARTITIONS REPLICAS RETENTION TIME COMPRESSION STATUS REASON
cat-facts-data computed 1 1 7days any resolution::provisioned
To verify, you can consume from the topic with the fluvio consume CLI.
We are using the -B option to start from the beginning offset of the topic. Once you reach the end of the topic, you can see new data as it is sent to the topic. To exit this live view, press Ctrl+C.
$ fluvio consume cat-facts-data -B
{"fact":"Female felines are \\superfecund","length":31}
{"fact":"Cats only sweat through their paws and nowhere else on their body","length":65}
{"fact":"While many parts of Europe and North America consider the black cat a sign of bad luck, in Britain and Australia, black cats are considered lucky.","length":146}
^C
When you want to stop the connector, you can delete it with fluvio cloud connector cat-facts
$ fluvio cloud connector delete cat-facts
connector "cat-facts" deleted
Deleting your connector will not delete the topic used by the connector. If you want to delete the topic, you can run fluvio topic delete cat-facts-data
$ fluvio topic delete cat-facts-data
topic "cat-facts-data" deleted
We created a basic Inbound HTTP connector, looked at the logs for the connector, and viewed the HTTP response data in the Fluvio topic. Lastly, we deleted the connector and topic.
You are ready to create your own connectors! Check out the docs for our supported Inbound and Outbound connectors to get started with your own data sources.
