Stambia versions 2.x, 3.x, S17, S18, S19 and S20 are reaching End of Support January, 15th, 2024. Please consider upgrading to the supported Semarchy xDI versions. See Global Policy Support and the Semarchy Documentation.
The Stambia User Community is moving to Semarchy! All the applicable resources have already been moved or are currently being moved to their new location. Read more…
Stambia DI can work with Twitter to perform operations such as posting and searching Tweets, retrieve various information, and more...
You will find in this getting started article the first steps to start working with Stambia DI Twitter Component.
You can also refer to Stambia DI for Twitter article for further information about supported operations, download page, and more.
Twitter is the most used social media platform to express views and tweet statuses. Gaining an insight on the data is one of the areas of interest for organizations. Getting data like the current trends, searching tweets, users etc. can provide very useful information.
Stambia Data Integration enables users to quickly get connected to Twitter API and use all the REST APIs made available by Twitter. Data can be easily pulled from and pushed to twitter using simple mappings.
Create an application on https://apps.twitter.com
Download Twitter API metadata from http://stambia.org
Templates Required – Generic & WSDL
Import the metadata packages downloaded from http://stambia.org into your project.
Right Click on the Project >> Import>>Archive File>>Browse the file>>Click OK
Once imported, you can double click on the metadata link to see the types of requests and their properties.
An additional step for Oauth configuration is required for authentication and authorization. The application created on https://apps.twitter.com will provides a Consumer Key and Consumer Secret. Using that, a Token Key & Secret would be created.
Right Click on the TwitterAPI, then Select Actions and click on Launch OAuth security wizard.
This will launch the wizard and you can provide the Consumer Key & Secret. Then Click on application selection button as shown below.
Select Twitter as the application.
Provide the Callback URL and then Click on Get Token!
You should see a popup from twitter app for authorization. Authorize app to generate the Token.
The Token and Token Secret is generated and configured in the wizard.
Browse through all the GET, POST & DELETE requests. View the properties and modify them if required. For e.g. below list of GET request can be reduced, by keeping just the request needed for a project and deleting everything else.
As well the metadata inside these requests can be viewed and modified. This would be a standard format of Input Parameters and Response JSON structure : -
Once the metadata is set-up, we can work on some mapping to pull data out of Twitter.
As an example, we can get the current trending topics in all the regions and then using their WOEID pull out top 50 trending topics and the volume of tweets for each topic : -
For this we need to use two requests : -
trendsAvailable – No Input Parameters required. Gives all the trending topics for a region and the region’s WOEID.
trendsPlaces – Input WOEID to get the top 50 trending topics and the tweet volume.
In a single mapping, we execute two GET requests connected to each other and get the data out of Twitter into a RDBMS staging table.
In cases where a huge amount of data must be moved to Hadoop for analytics, Hadoop templates can be used to push this data. The data can be processed in Spark and results can be put in a Hive Table.
Use search Tweets Request to search keywords used in the tweets and then move the data to HDFS.
Once the file has been moved to HDFS, this can be processed in Spark.
Additionally, you can use the other request from Twitter like POST, DELETE etc.
Make sure you have the Read and Write permission on the Token.
Stambia Data Integration allows to work with Twitter.
You can reverse in Metadata the common operations which can be performed with Twitter such as posting and searching Tweets, retrieve various information, and more...
You can also design your own operations as you want.