ABSYZ ABSYZ

  • Home

    Home

  • About us

    Who We Are

  • Our Expertise

    What we Do

  • Our Approach

    How We Do It

  • Products

    What We Made

  • Industries

    Who We Do It For

  • Clients

    Whom We Did It For.

  • Article & Blogs

    What Experts Think

  • Careers

    Join The Team

  • Get In Touch

    Let’s Get Started

ABSYZ

BULK API 2.0

Home / Article & Blogs / Articles / BULK API 2.0
By Pavankumar Gudavalli inArticles

Introduction to BULK API 2.0

Bulk APIs are used in salesforce wherever there is a necessity to process large amounts of data. Usually, any data processing involving more than 2000 records should be an ideal situation to use bulk API. These APIs can be used to query, insert, update, upsert and delete Salesforce data. Bulk API 2.0 is more streamlined and simpler when compared to 1.0.

BULK API 2.0 query workflow example

BULK API workflow is more complex.

Bulk APIs are based on REST principles and work asynchronously. When a query request is made these bulk API jobs automatically find the best way to divide the entire query result into chunks and send back each chunk. This avoids timeouts and gives you time to process the current chunk before the next chunk comes in(Detailed example below).

While ingesting data, salesforce creates a batch for every 10000 records and the maximum number of records allowed in a day is 150,000,000. Please find full details on how requests are processed here.

Let’s jump into a use case where we can use BULK API 2.0, how to request, process the data.

Use Case:

Assume you wanted to download all the opportunity records in the org as your company needed this information to process in an external system. Let’s deep dive into this.

The technical challenges involved here are opportunity records are in lakhs and you really can’t query all records by using normal SOQL query from the apex in one go, even though you use limit and offset, the maximum offset value that can be used is 2000 and any offset more than that gives NUMBER_OUTSIDE_VALID_RANGE error. Let’s solve all of these issues using bulk API 2.0.

Let’s start by creating a bulk API query call. We will first generate the call to retrieve data and download it later. Here is an example of how you can make this request.

HttpRequest Request= new HttpRequest();
Request.setMethod(‘POST’);
Request.setEndpoint(URl.getOrgDomainUrl().toExternalForm()+’/services/data/v50.0/jobs/query’);
Request.setHeader(‘content-type’, ‘application/json’ );
Request.setHeader(‘Authorization’ ,’Bearer ‘+userInfo.getSessionId());
String str_body = ‘{ ‘+
‘”operation”: “query”,’+
‘”query”: “select  Name, Id from Opportunity “,’+
 ‘”contentType” : “CSV”,’+
  ‘”lineEnding” : “CRLF”‘+
 ‘}’;
 Request.setBody(str_body );
Http http1 = new Http();
HttpResponse Response= http1 .send(Request);

Headers in the request should include Authorization and content-type headers. Authorization becomes easier as this request is made inside the salesforce. Just logged in user session-id will do the trick. This authorization will be different when these requests are made outside of the salesforce.

The body of the request should include content type (currently only CSV is supported), the operation should be “query” since we are retrieving data and the query includes the SOQL query we want to perform. LineEnding crlf mean Carriage Return which indicates end the line with ‘\n’.

Response to this above request will be similar as below.

This request will not contain any data but it gives details about the asynchronous job. Data can be retrieved in the second step.

Response in this request includes a job Id which we have to store in order to retrieve data.

Map<String, Object> Map_response = (Map<String, Object>) Json.deserializeUntyped(Response .getBody());
String str_jobId = (String)Map_response .get(‘id’);

This request starts retrieving data in the backend but we have to perform another request to bring that data, which is our second step.

This second step involves making another HTTP request to retrieve data.

HttpRequest DataReq = new HttpRequest();
DataReq .setMethod(‘GET’);
DataReq .setEndpoint(URl.getOrgDomainUrl().toExternalForm()+’/services/data/v50.0/jobs/query/’+str_jobId +’/results?maxRecords=5000′);
DataReq .setHeader(‘content-type’, ‘text/csv’ );
DataReq .setHeader(‘Authorization’ ,’Bearer ‘+userInfo.getSessionId());
Http Http2 = new Http();
HttpResponse DataRes = Http2 .send(DataReq );
String str_ CSVBody = (String)DataRes .getBody();
String str_locator = (String)DataRes.getHeader(‘Sforce-Locator’);

The response body in the above request includes 5000 opportunity records in the CSV format which we are assigning to a string, along with headers which are field names in the query we requested. Now the request gives you only 5 records since in the request we specified it to retrieve only 5000 (maxRecords=5000).

Remember the lower value you give in maxRecords the more number of requests it takes to finish the entire request. There is another catch here, if you specify a maximum number of records to a very high number with more number of fields in the query there is a good chance timeout error can be encountered. So this number should be in a sweet spot where it should not be very high or low.

Since maxRecords values vary from one scenario to another. No finite range for this value.

In the response, one of the headers also includes “’Sforce-Locator’” which acts as OFFSET. Now in order to retrieve more records we have to use this Sforce-Locator.

HttpRequest DataReq2 = new HttpRequest();
DataReq2.setMethod(‘GET’);
DataReq2.setEndpoint(URl.getOrgDomainUrl().toExternalForm()+’/services/data/v50.0/jobs/query/’+jobId+’/results?locator=str_locator &maxRecords=5000′);
DataReq2.setHeader(‘content-type’, ‘text/csv’ );
DataReq2.setHeader(‘Authorization’ ,’Bearer ‘+userInfo.getSessionId() );
Http http3 = new Http();
HttpResponse DataRes2= http3.send(DataReq2);
str_CSVBody = (String)DataRes2.getBody();
str_locator = (String)DataRes2.getHeader(‘Sforce-Locator’);

Now when you use the locator from the second request it gives you another 5000 records continued from the first request. In other words, it gives you records from 5000 to 10000.

Since this HTTP request is considered a callout, consider callout limits.

Now similarly you will get this salesforce data until all your query resultant records are retrieved in chunks of 5000. Remember if you are using aura or lwc for this, after a certain point your page will become idle because when the data response comes in we are assigning that chunk of 5000 records to a string str_CSVBody  and in the next callout we will assign another 5000 records to the same string. We are doing this because our ultimate goal is to download these records. Strings can hold data up to a certain limit and when you are dealing with lakhs of records we might hit that.

So a better practice would be to keep a count of the number of records that are added to the string and download the file after it reaches a certain number. Please find below the code to download records in the CSV file.

var blob  = new Blob([“\ufeff”,str_CSVBody]);
var url = URL.createObjectURL(blob);
var downloadLink = document.createElement(“a”);
downloadLink.href = url;
downloadLink .download = ‘OpportunityRecords.csv’;
downloadLink.click();
document. body.removeChild(downloadLink);

Reset the string after downloading and repeat the same. You might end up with multiple files but you will have all the records.

This example is for querying the records, for inserting the records endpoint URL changes to ‘/services/data/v50.0/jobs/ingest/’. Request body also changes. You can find details here.

Bulk API also has certain restrictions. Please find the entire limits here.

Link for more details regarding BULK API 2.0 –  Click here





bulk api 2.0bulk api 2.0 querybulk api 2.0 salesforcebulk api 2.0 trailheadsalesforce bulk api 2.0
383
Like this post
1 Post
Pavankumar Gudavalli

Search Posts

Archives

Categories

Recent posts

BioAsia 2023 in Hyderabad: An Annual International Event

BioAsia 2023 in Hyderabad: An Annual International Event

The Role Of Marketing in Small & Medium Enterprises

The Role Of Marketing in Small & Medium Enterprises

Salesforce For Retail: How Salesforce CRM Can Help Retailers

Salesforce For Retail: How Salesforce CRM Can Help Retailers

What is ChatGPT & How Does It Work?

What is ChatGPT & How Does It Work?

What Is Graphic Design? (Executive Summary 2023)

What Is Graphic Design? (Executive Summary 2023)

  • Migrate to Flow Tool (Beta) - Salesforce Spring '22 Release
    Previous PostMigrate to Flow Tool (Beta) - Salesforce Spring '22 Release
  • Next PostABSYZ focuses on a modern Employee Training and Development Plan for attaining great goals
    Migrate to Flow Tool (Beta) - Salesforce Spring '22 Release

Related Posts

Dreamforce 2021 – Takeaways
Articles

Dreamforce 2021 – Takeaways

US Independence – a tribute by Zaheer Arif
Articles

US Independence – a tribute by Zaheer Arif

Significance of UI/UX Design
Articles

Significance of UI/UX Design

The world of AR and VR
Articles

The world of AR and VR

Leave a Reply (Cancel reply)

Your email address will not be published. Required fields are marked *

*
*

ABSYZ Logo

INDIA | USA | UAE

  • About us
  • Article & Blogs
  • Careers
  • Get In Touch
  • Our Expertise
  • Our Approach
  • Products
  • Industries
  • Clients
  • White Papers

Copyright ©2022 Absyz Inc. All Rights Reserved.

youngsoft
Copy
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “ACCEPT ALL”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent. Privacy Policy
Cookie SettingsREJECT ALLACCEPT ALL
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled

Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.

CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Functional

Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.

Performance

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Analytics

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.

Advertisement

Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.

Others

Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.

SAVE & ACCEPT