[SOLVED] [Javascript] Give new image data to Sprite (from websocket stream)

How do I...?
1

I am trying to replace a sprites image (no animation, single frame) with jpeg video data received over a websocket (as received in a javascript code block).

Below is the data receiving function of the WS code, and included for reference is how it worked on CANVAS, but I can’t make it work in Gdevelop/PIXI - even using the PIXI.Sprite.fromImage(imageUrl) seems to do nothing, unless i am assigning it to the wrong object.

So mabye if smoeone can tell me which object i can assign it to that would be awesome ? :S I just need to update a sprite with some received video frames in jpeg format.

// code example

function onWebsocketMessage(evt) {
    var arrayBufferView = new Uint8Array( evt.data );
    var blob = new Blob( [ arrayBufferView ], { type: "image/jpeg" } );
    var urlCreator = window.URL || window.webkitURL;
    var imageUrl = urlCreator.createObjectURL( blob );
    console.log("video frame received");
   
    // obviously the below is incorrect, this is how it works in a CANVAS html5 app, however i can not figure a way to output the imagedata (which is in a string array from above, as well as into a img.src compatible blob url) into a PIXI sprite.
    // I have tried gaining the objects[0].parent to get access to PIXI.Sprite and using PIXI.Sprite.fromImage(imageUrl); but any assignment i make just does nothing silently (nothing in the js console).

    img.onload = function(e) {
        console.log("\tdraw image size: " + e.target.width + "x" + e.target.height);
        contextVideo.drawImage(img,vPosX*xScale,vPosY*yScale,380*xScale,280*yScale);
        fpscounter=fpscounter+1;
    };
    img.src = imageUrl;
}
2

This example can help you :wink:

Editor online : Load image from url

2 Likes
3

I got it working by inspecting some other slightly-related examples here on the forum!

Here is the working code-block incase it helps anyone else one day;

function onVideoMessage(evt) {
    var arrayBufferView = new Uint8Array( evt.data );
    var blob = new Blob( [ arrayBufferView ], { type: "image/jpeg" } );
    var urlCreator = window.URL || window.webkitURL;
    var imageUrl = urlCreator.createObjectURL( blob );
    console.log("video frame");
    
    var img = new Image();
    img.onload = function(e) {

        var object_texture_image = runtimeScene.getObjects("videobox");
        var object_texture_image_renderer = object_texture_image[0].getRendererObject(); 
        object_texture_image_renderer.texture = new PIXI.Texture((new PIXI.BaseTexture(img)));
    };
    img.src = imageUrl;
}
1 Like
4

You are awesome - i actually found that example and used it to make it work before even seeing your reply! :slight_smile: you’re too fast! haha thanks!

1 Like
5

You use video stream in your code with WS inside GD ? i’am very curious to see that :slight_smile: !

1 Like
7

Thanks to you mate!

The working example (very basic, no error handling etc, just a ‘test’ to see if i could get my stream into a GD app):

published test here: http://gd-games.s3-website-eu-west-1.amazonaws.com/game-1566606660918/

project package + code here;
http://saltybrains.com/gd-ws-video-example/gd-ws-test.zip

thanks again!

8

Next I want to do some tests to figure out a way to incorporate JSMpeg with Websocket stream delivery to get audio+video and better codec into it. I have working examples in canvas, and if i can bring those basics over into GD then i’ll start using GD exclusively for my new gaming products (live casino games), as the rest of it seems pretty simple in this.

I’ve only been using GDevelop for a day, and I really like it so far, I can see this being very fast for game production.

9

Instead use sprite object, prefer use video object, it can provide a better manage of your video stream.
Prefer also use H.264/mp4 and AAC on audio :wink:

Nice thank you for this example :slight_smile:

10

yeah but the video object doesn’t look like it would work right, firstly it wants a locally available video file but even without, i need live realtime streaming and typically those widgets wont work. I need 100% control of the stream and process, unfortunately.

11

Thanks again for your help here and on other posts - I have it working now with JSMpeg (modified with some tweaks including inter-frame metadata and a CDN structure (source->core->relay) and the quality is great! a huge improvement!

An early sample of the game can be seen here: http://relay.saltybrains.com/test/

1 Like
12

look nice, you are doing a lot of work with JS, hope you share something in the future :wink:

1 Like
13

Thanks, and sure - what would you like to know ? i’ll share code of whatever you need.

The latest thing I finally figured out was tweening from JS: I managed to integrate a 3rd party TWEEN library to GD, so that i can issue tweens in JS code like this below which handles the ‘bouncing chips’ test in the above preview;

This code gives a ‘callback’ when the tween completes also. “obj” refers to any object you give it, such as in a loop or otherwise (eg: object[0] ).

var coords = { x: obj.getX(), y: obj.getY() };
    var tween = new runtimeScene.TWEEN.Tween(coords) 
	.to({ x: obj.getX(), y: obj.getY()-20 }, 300+(50*chipnum)) 
	.repeat( Infinity )
	.yoyo( true )
  .easing(TWEEN.Easing.Elastic.InOut) 
	.onUpdate(function() { // Called after tween.js updates 'coords'.

		obj.setX(coords.x);
		obj.setY(coords.y);
	})
	.onComplete(function(){
		console.log("tween complete");
	})
	.start();
1 Like
14

I don’t know everything you do sounds good :smile: . The effect of the coins is really nice.
Only would be good if your work could be resused like extensions for other people, that would make gdevelop better.

15

Thanks! - yeah I would love to make some extensions that can be used but I haven’t really figured that out yet - just setting up the functions that I need to pull my game together, testing a way to accomplish each part as I go - if I figure out a way to add easily useable modules I’d love to give them out but so far I can’t see a nice way to do that… it seems like a bit of a bandaid to just copy-paste code blocks in… but maybe that’s the way to do it.

Though these posts on this forum are useful, the ones i’ve read have helped me and hopefully the ones that i’ve asked that are now marked as ‘solved’ will help others, as i’ve tried to clearly show what/how it was done for anyone else who wants to do similar.

I’ve only been on GD for a week, still just figuring it out until I can get one project done :slight_smile:

2 Likes