Have you ever been in a situation where you need to store a gigantic amount of data in your Salesforce org? Are you tired of using third-party storage systems and writing web service calls to get the massive amount of data? Having to crunch those large numbers, though a reflection of your own success, can be troublesome and performance degrading. Salesforce has come to the rescue again by introducing big objects-which are objects with massive storage capabilities on the Salesforce platform itself. They provide consistent performance over a data set of the order of billions and are accessible through a standard set of APIs to your org or to an external system.

What exactly are Big Objects?

Big Objects are merely custom objects but with massive data handling capabilities. They can store at least billions of data in them. They are similar to custom objects in some ways but very different in others. The scale of data is what lets Big objects stand apart in the crowd.

Just like normal Salesforce objects, Big Objects are also standard as well as custom. An example of a standard big object is FieldHistoryArchive, which stores archived field history data for Field Auditing purposes. Custom big objects are created and deployed by you using the Metadata API.

When can you use a Big Object?

As a massive data store, Big Objects can be a great help in many cases. Some typical uses of Big Objects are as follows:

1. Audits and Tracking purposes
2. Better customer experiences
3. Historical Data Archive

How to create a Big Object?

As gigantic of a task, it may seem at first, creating a big object is not that difficult. Obviously, you do not get an interface like that in a custom object, but the extra work is worth it.
There are mainly three files associated with creating a big object:

1. Object
2. Permission Set
3. XML

As we know, All work but no play makes Jack a dull boy, let’s play around with creating a big object for ourselves. Let it be called Test Big Object.

1. Object

This is a file with the .object extension and is named on the API name of the Big Object. It contains the fields and indexing details about the big object. The file contains data in XML format.
The Test_Big_Object__b.object file will look like this,
<?xml version=”1.0″ encoding=”UTF-8″?>
 <CustomObject xmlns=”http://soap.sforce.com/2006/04/metadata”>
<deploymentStatus>Deployed</deploymentStatus>
<fields>
<fullName>Test_Field_One__c</fullName>
 <label>Test Field One</label>
<required>false</required>
<type>Number</type>
<unique>false</unique>
 <precision>7</precision>
<scale>4</scale>
</fields>
<fields>
<fullName>Test_Field_Two__c</fullName>
<label>Test Field Two</label>
<length>16</length>
<required>false</required>
 <type>Text</type>
 <unique>false</unique>
</fields>
<indexes>
<fullName>Test_Big_Object_Index</fullName>
 <label>Test Big Object Index</label>
 <fields>
  <name>Test_Field_One__c</name>
<sortDirection>DESC</sortDirection>
 </fields>
</indexes>
<label>Test Big Object</label>
 <pluralLabel>Test Big Object</pluralLabel>
</CustomObject>

2. Permission Set

This file contains the permissions on each field defined in the big object. This file is also in XML format but has the extension .permissionset.This file is not mandatory to include in your package, but you need it to give access to users since a big object permission is restricted by default.
The Test_Big_Object.permissionset will look like this,
<?xml version=”1.0″ encoding=”UTF-8″?>
<PermissionSet xmlns=”http://soap.sforce.com/2006/04/metadata”>

 <fieldPermissions>
 <editable>true</editable>
<field>Big_Object__b.Test_Field_One__c</field>
 <readable>true</readable>
</fieldPermissions>
<fieldPermissions>
<editable>true</editable>
<field>Big_Object__b.Test_Field_Two__c</field>
 <readable>true</readable>
</fieldPermissions>
<label>Test Big Object Permission Set</label>
</PermissionSet>

3. Package.xml

This file is like any other package.xml file. It only specifies the contents of the package. The package.xml file for our Test Big Object will look like this,
<?xml version=”1.0″ encoding=”UTF-8″?>
<Package xmlns=”http://soap.sforce.com/2006/04/metadata”>
<types>
 <members>*</members>
 <name>CustomObject</name>
</types>
<types>
  <members>*</members>
 <name>PermissionSet</name>
 </types>
 <version>41.0</version>
</Package>

Now that we have all our files in place, there is a particular structure of the package that you need to follow. First, the package should be in zip format. The package should contain 2 folders- Objects and Permissionsets. The Test_Big_Object__b.object file goes into the objects folder while the Test_Big_Object.permissionset file goes in the permissionsets folder. The package.xml file will reside in the root folder itself.

You can deploy your big object package using Workbench. While deploying, select the Single Package Only checkbox and leave other options untouched. Choose your package and deploy it. Now, to check whether your big object has been created or not, go to Setup  Big Objects. Your big object will be listed there.
Viola! You just created your first big object.

How to query a Big Object?

Obviously, keeping all that massive data in your org would be a waste if you can’t process it. Well, you can. To query the data stored in a big object, you can use a standard SOQL query or Async SOQL.

Using SOQL to query big objects has a disadvantage of large waiting times if the returned data set is large. Hence, it is advisable to use standard SOQL only if the query will return a small data set. For returning large sets of data, we must use Async SOQL. Now as is evident from the name, Async SOQL runs asynchronously. It is a way to run SOQL queries in situations where you can’t wait for the results in real-time due to the massive data that is being queried.

Async SOQL features:

• Highly scalable solution with the ability to process billions of data
• Ability to run multiple queries in the background while continuously monitoring their status
• Easy to use for someone coming from a SOQL background due to usage of a subset of SOQL commands

Limitations

As limitless as they may seem; just like every other feature present in Salesforce, Big Objects have their limitations.

1. You can create up to 100 big objects in your org. Although, this can vary depending on the license you have.
2. Given the large scale of data they operate on, you cannot use triggers, workflows, processes, and the Salesforce app with big objects.
3. Well, no UI or pointing & clicking here. You can only create big objects through the Metadata API.
4. For the creation of records, you must use Apex or bulk DML operations. No UI is present for record addition.