Amazon is making a bigger leap into open-source technology with the unveiling of its machine-learning software DSSTNE.
The newly released program is competing with Google’s TensorFlow, which the search giant open-sourced last year. Amazon says DSSTNE (which stands for Deep Scalable Sparse Tensor Network Engine and is pronounced “Destiny”) excels in situations where there isn’t a lot of data to train the machine-learning system, whereas TensorFlow is geared for handling tons of data.
DSSTNE is also faster than TensorFlow, with Amazon claiming up to 2.1 times the speed in low-data situations. The software comes from Amazon’s need to make recommendations in its retail platform, which required the company to develop neural network programs. However, Amazon doesn’t always have a lot of data to work from when making those recommendations.
Amazon’s system can achieve those speeds in part due to multi-GPU capabilities. Unlike other open-source “deep learning” programs, DSSTNE can automatically distribute its workload across many GPUs without speed or accuracy tradeoffs that often come with training across multiple machines.
“This means being able to build recommendations systems that can model ten million unique products instead of hundreds of thousands,” the DSSTNE FAQ says. “For problems of this size, other packages would need to revert to CPU computation for the sparse data, decreasing performance by about an order of magnitude.”
Amazon is releasing the software as an open-source project to help machine learning grow beyond speech and language recognition that many companies are focusing on, instead expanding into areas like search and recommendations.
“We hope that researchers around the world can collaborate to improve it,” the FAQ reads. “But more importantly, we hope that it spurs innovation in many more areas.”
DSSTNE is available now on GitHub under an Apache 2.0 open-source license.
